Optimize remote logging in vCenter v6.7

I’m sure many of the companies are using Splunk as centralized log server, and are aware of the licensing method of it. As a Splunk customer you will pay for the amount of logs received by your Splunk environment which could be painful for many large companies, and still, Splunk is a very popular product.
I have to say, I understand that someone has to pay the bill, and usually the decision was made by the “business” instead of the IT professional, who works on the systems. I’ve met a couple employees who had the ongoing task to optimize the outgoing logs from the environment.

And they have no choice, somehow the outgoing logs must be filtered. It doesn’t matter if you don’t have the necessary information stored centrally, or you may have to log into the systems one by one which can be time consuming, when you have to analyse logs in a large environments. Aim is to decrease the outgoing amount.

This is why this article has been created. I’ll try to collect the options available in a basic vSphere environment to decrease the amount of outgoing logs.

In vCenter GUI the following log levels are available:

Log LevelDescription
None (Disable logging)Turns off logging
Error (Errors only)Displays only error log entries
Warning (Errors and warnings)Displays warning and error log entries
Info (Normal logging)Displays information, error, and warning log entries
Verbose (Verbose)Displays information, error, warning, and verbose log entries
Trivia (Extended verbose)Displays information, error, warning, verbose, and trivia log entries

Caution: Trivia and Verbose mode should be used for troubleshooting and for a limited time only. After the troubleshooting is done, change back to the default!

vCenter

In vCenter server you can choose easily from the above options if you click on Hosts and Clusters tab › vCenter name › Configuration › Settings › General › Edit

Alternatively the same setting is available on the advanced settings tab:

If it’s not enough, and there are logs still to be filtered, rsyslog can be modified as well.

Check that the central syslog server is added on the VAMI interface. Go to the https://vCenterIP:5480 and log in. Select the syslog tab and configure your syslog server.

As next step open a cli to the vcenter ip and check the /etc/vmware-syslog/syslog.conf.
It should look like this:

*.* @192.168.1.1:514;RSYSLOG_SyslogProtocol23Format

This needs to be edited according to the requirements:

*.warn*.err;*.crit;*.alert @192.168.1.1:514;RSYSLOG_SyslogProtocol23Format

Once it’s done, restart rsyslog service

systemctl restart rsyslog

You can test the result with logger:

logger -p syslog.info "Info level / this will not appear"
logger -p syslog.warn "Warning level / this will be visible"
logger -p syslog.error "Error level / this will be visible"

If you have external PSC(s) this should be done on that appliance(s) as well.

Note: If something changed on VAMI interface, this configuration may require to be added again!

Upgrade Manager

To change Update Manager’s log level we need the old flex gui. Click on Home  > Administration > System Configuration > Services > Update manager > Manage.

That’s all. I hope this guide provides enough info to keep the unwanted logs away from your syslog servers. Having said all this: information is important and with log filtering only to error messages you may not have that visibility of your environment that you and your monitoring, reporting tool would need for proper analysis. If you implement log filtering you will be notified only if the issue happened and pro-active intervention becomes difficult.

Many other solutions exist and worth considering instead of Splunk. For example: VMware Log Insight using OSI (single log source) model or per CPU basis, where there is one flat fee for all log data sources from a single CPU, regardless of hypervisor or number of guest OSs.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s