Monitoring Using ELK

Logging , monitoring and Alerting  are the key components in a software life cycle. In the previous blog we discussed some of the best practices of logging. Let’s have a discussion on monitoring & alerting using ELK. 

 As you all know sometimes unexpected things can cause major issues. You will try to solve the problem in a belief that it will resolve all the issues. But at the same time it can also make things more complicated. Especially when you deal with micro services. When you are moving from a monolithic architecture to a micro service architecture it includes more flexibility and faster deployment.  Thus we realize the importance of an effective tool which can improve the performance of the system which boosts productivity and reduces downtime. 

 This is where the ELK stack plays its role and help you to collect the data from all the resources and through that you can search and analyze the data you need when and where it required


ELK Stack or Elastic Stack is a trending topic in Log Management

What is ELK Stack

 If you are looking for a simple explanation,  I will say ELK stack is the combination of three open source tools. Elasticsearch, Logstash & Kibana. When these three combine together and form a tool/platform for the log management system. It allows you to search all the logs in a single platform. It also simplifies to find out the issues in multiple servers, connecting logs within a specific time frame. 


 Elasticsearch is basically a search engine. A highly scalable search engine which runs on top of Java-based Lucene engine. Elasticsearch allows you to store, search and analyze huge volumes of data so quickly and in real time which gives back your search results in a fraction of seconds. 

 Some features & advantages of Elasticsearch

  • It is an open source search server written in JAVA
  • Store schema less data and also creates a schema for  your data
  • Based on APACHE Lucene and provides RESTful API
  • Provides horizontal scalability reliability and multi tenancy capability  for real time  use of indexing  to make it faster  search 
  • Enables near real time search which helps you to  scale vertically and horizontally


 In order to search data we need to collect the data first.  logstash is the data collection pipeline tool. It is the tool which collects data and feeds into Elasticsearch.

.Some features & advantages of Logstash

  • Logstash allows different inputs for your logs
  • It offers centralized data processing 
  • Another major advantage is it allows plugins to connect with various types of input sources and platforms. 


 Kibana is an open source visualization and analytical tool which completes the ELK stacks . Main use is to visualize the Elastic search documents and helps developers to have a quick awareness of it. Kibana simplifies the huge volumes of data and reflects the real time changes in the Elasticsearch queries.

 Kibana’s main features and advantages are

  •  Enable real time search of indexed information 
  • Fully integrated with Elasticsearch
  • Kibana can provide historical data in the form of graphs and charts. 
  • It is capable of sharing the snapshots of the logs searched


 As you all know while receiving, transforming and sending data, there can be performance issues. Here comes the role and its importance of Beats. Beats is a free and open platform for single-purpose data shippers. They send data from thousands of machines and systems to Logstash or Elasticsearch. 

 Beat is a collection of agents  that are installed on your system to collect a specific set of information . There are many types of Beats  and currently 8 of them are important. Each Beats has a specific function. Each beat can collect various types of data. for eg: there is Filebeats for log files, Functions Beats for server less cloud infrastructure, Metricbeats for system metrics and Packetbeats for network Packets. 

 This note will be incomplete without discussing the challenges we face while using ELK Stack . It is indisputable that ELK Stack is a wonderful solution for managing huge data and log in short time. Even Though, whenever you move on to a complex setup, different components of the stacks can become difficult to handle.

Remember there is nothing like Trial and Error. Thus the more you do the more you learn along the way. 

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>