Install Log Insight Multiple Nodes and integrate with KEMP Load Balancer for HA

This post has already been read 8834 times!

VMware vCenter Log Insight is a VMware analytics product introduced one year ago. Part of VMware vCenter family, Log Insight delivers automated log management through log analytics, aggregation, and search, extending VMware’s leadership in analytics to log data. The new Log Insight 2.0 version is much faster, more effective and usable:

  • 8X faster data collection
  • 6X query performance But also able to scale up and out in several ways:
  • Increase nodes (up to 6) to an existing Log Insight installation.
  • 2TB of live searchable data per node.
  • High availability: no single point of failure for log ingestion.
  • Load balanced via external load balancer. more about this in this post.
  • Single UI for distributed queries and single management interface.

VMware Log Insight supports receipt and ingestion of Syslog messages that are sent over UDP, TCP, TCP with SSL encryption and via API. I’m going to be using this in the lab to collect all the syslogs from VMware vSphere hosts and Windows VM’s thru agent. image

What I wanted to try is to have my lab log to the load balancer and distribute the logs to multiple Log Insight nodes. Like in the picture above. Great for the bigger environments.

Deploy two or more Log Insight nodes (workers)

I deployed two new Log Insight Appliance in my Lab. Both Extra Small Configuration (2CPU / 4Gb memory). Configured a static IP and make sure the disk is Thick Eager Zeroed (much faster writes). Browse to the Log Insight website: https://<ip-address>. Follow the initial setup website and set admin password, e-mail adress, relay (if you have) and Finish the setup.
image
After the Initial setup there are no logs imported. Let’s proceed to install a second node where we choose to Join the first installed node. enter the FQDN of the first Log Insight node you installed.
image
To complete the process, you will need to access the Cluster Management page on the master and authorize this worker to join. Worker IP: <ip-address worker node> Add as many worked nodes as you want. Everything is administrated through the first node.

Install KEMP Load Balancer

We are going to install one KEMP Load Balancer to have a single point of entry for all logs. You can setup 2 load balancers to have HA features if you wish. (HA config document)

  1. Go to http://kemptechnologies.com/server-load-balancing-appliances/virtual-loadbalancer/vlm-download and download the appropriate KEMP Virtual LoadMaster.
  2. Follow the prompts to create a KEMP ID after the download begins.
  3. Import the VLM OVF into your VMware Infrastructure.
  4. Follow the instructions in the Licensing Feature Description document.
  5. I created new DNS records to access the load balancer thru FQDN.

image
I have just created a HA pair of 2 load balancers and a 3th DNS record syslog for the shared ip.

Update LB and Configure KEMP for Log Insight

The Log Insight add-on pack is required and this can be acquired by posting a General request in the KEMP Help Center Community: https://support.kemptechnologies.com/hc/communities/public/topics.
It will also be available for direct download later in September from the tools section of KEMP’s website at http://kemptechnologies.com/loadmaster-documentation#toolsSection

To install the Log Insight Add-On on the Virtual LoadMaster, please follow the listed steps: 1- Navigate back to System Configuration > System Administration > Update Software. Browse to the ‘addon’ file and click on “Install Addon Package”. clip_image006 2- Click “OK” on the resulting dialog box 3- Navigate to System Configuration > System Administration > System Reboot. Click on “Reboot”.

NOTE: Question marks in the top ribbon will indicate that you’ve lost access and the VLM is rebooting. Don’t click “Continue” so that the console automatically reloads upon completion of reboot.

4- Navigate back to System Configuration > System Administration > Update Software. You should now see that the “Log_Insight” package is set to 7.1-19-536. clip_image011

Configure the Load Balancer

Download this LoadMaster Deployment Guide – VMware vCenter Log Insight Manager​ document. A number of Virtual Services will need to be created for the LoadMaster to work effectively with Log Insight.

Refer to the downloaded document from section 2.2 for detailed, follow the step-by-step instructions to fully configure the Load Balancer.

The MOST important value of the solution comes from the fact that you can get even distribution across the cluster of Log Insight nodes and this is not possible natively anytime syslog is sent over any other transport than UDP.

The features that the add-on pack enables is a special service type called “Log Insight”:clip_image002

also check out the setting called “Log Interval Split” that controls how many messages should be directed to 1 server before moving to the next

clip_image004

Results

We can see all the syslog output send to both Log Insight Nodes. Spreading the load exactly 50%.image

We can see all the real servers (Log Inside workers) online. image

All events are distributed evenly on the Log Insight Nodes.

clip_image001

My 2ct

This setup is a great affordable way of a building a great Enterprise Log Analytics environment that can massively scale and is also High Available. I love the Log Insight Content Packs. Check it out for yourself!

About KEMP Technologies
KEMP LoadMaster™ is an advanced Layer 4-7 load balancing application delivery controller (ADC).  Flexible deployment options are available on a wide array of hypervisor and cloud platforms, as well as dedicated KEMP appliances and third party best-in-class ‘bare metal’ servers. LoadMaster provides enterprise application integration and acceleration services that intelligently direct user traffic to makes applications perform better in on-premise, virtualized and hybrid cloud data center architectures.

About Marco Broeken

Marco Broeken is Author of this blog and owner of vSpecialist Consulting and has 20 years experience in IT. Marco has been rewarded with the vExpert status from 2011 - 2018.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.