This post has already been read 11053 times!
VMware vCenter Log Insight is a VMware analytics product introduced one year ago. Part of VMware vCenter family, Log Insight delivers automated log management through log analytics, aggregation, and search, extending VMware’s leadership in analytics to log data. The new Log Insight 2.0 version is much faster, more effective and usable:
- 8X faster data collection
- 6X query performance But also able to scale up and out in several ways:
- Increase nodes (up to 6) to an existing Log Insight installation.
- 2TB of live searchable data per node.
- High availability: no single point of failure for log ingestion.
- Load balanced via external load balancer. more about this in this post.
- Single UI for distributed queries and single management interface.
VMware Log Insight supports receipt and ingestion of Syslog messages that are sent over UDP, TCP, TCP with SSL encryption and via API. I’m going to be using this in the lab to collect all the syslogs from VMware vSphere hosts and Windows VM’s thru agent.
What I wanted to try is to have my lab log to the load balancer and distribute the logs to multiple Log Insight nodes. Like in the picture above. Great for the bigger environments.
Deploy two or more Log Insight nodes (workers)
I deployed two new Log Insight Appliance in my Lab. Both Extra Small Configuration (2CPU / 4Gb memory). Configured a static IP and make sure the disk is Thick Eager Zeroed (much faster writes). Browse to the Log Insight website: https://<ip-address>. Follow the initial setup website and set admin password, e-mail adress, relay (if you have) and Finish the setup.
After the Initial setup there are no logs imported. Let’s proceed to install a second node where we choose to Join the first installed node. enter the FQDN of the first Log Insight node you installed.
To complete the process, you will need to access the Cluster Management page on the master and authorize this worker to join. Worker IP: <ip-address worker node> Add as many worked nodes as you want. Everything is administrated through the first node.
Install KEMP Load Balancer
We are going to install one KEMP Load Balancer to have a single point of entry for all logs. You can setup 2 load balancers to have HA features if you wish. (HA config document)
- Go to http://kemptechnologies.com/server-load-balancing-appliances/virtual-loadbalancer/vlm-download and download the appropriate KEMP Virtual LoadMaster.
- Follow the prompts to create a KEMP ID after the download begins.
- Import the VLM OVF into your VMware Infrastructure.
- Follow the instructions in the Licensing Feature Description document.
- I created new DNS records to access the load balancer thru FQDN.
Update LB and Configure KEMP for Log Insight
The Log Insight add-on pack is required and this can be acquired by posting a General request in the KEMP Help Center Community: https://support.kemptechnologies.com/hc/communities/public/topics.
It will also be available for direct download later in September from the tools section of KEMP’s website at http://kemptechnologies.com/loadmaster-documentation#toolsSection
To install the Log Insight Add-On on the Virtual LoadMaster, please follow the listed steps: 1- Navigate back to System Configuration > System Administration > Update Software. Browse to the ‘addon’ file and click on “Install Addon Package”. 2- Click “OK” on the resulting dialog box 3- Navigate to System Configuration > System Administration > System Reboot. Click on “Reboot”.
NOTE: Question marks in the top ribbon will indicate that you’ve lost access and the VLM is rebooting. Don’t click “Continue” so that the console automatically reloads upon completion of reboot.
4- Navigate back to System Configuration > System Administration > Update Software. You should now see that the “Log_Insight” package is set to 7.1-19-536.
Configure the Load Balancer
Download this LoadMaster Deployment Guide – VMware vCenter Log Insight Manager document. A number of Virtual Services will need to be created for the LoadMaster to work effectively with Log Insight.
Refer to the downloaded document from section 2.2 for detailed, follow the step-by-step instructions to fully configure the Load Balancer.
The MOST important value of the solution comes from the fact that you can get even distribution across the cluster of Log Insight nodes and this is not possible natively anytime syslog is sent over any other transport than UDP.
also check out the setting called “Log Interval Split” that controls how many messages should be directed to 1 server before moving to the next
All events are distributed evenly on the Log Insight Nodes.
This setup is a great affordable way of a building a great Enterprise Log Analytics environment that can massively scale and is also High Available. I love the Log Insight Content Packs. Check it out for yourself!
About KEMP Technologies
KEMP LoadMaster™ is an advanced Layer 4-7 load balancing application delivery controller (ADC). Flexible deployment options are available on a wide array of hypervisor and cloud platforms, as well as dedicated KEMP appliances and third party best-in-class ‘bare metal’ servers. LoadMaster provides enterprise application integration and acceleration services that intelligently direct user traffic to makes applications perform better in on-premise, virtualized and hybrid cloud data center architectures.