Speakers
Description of Work
As a distributed computing environment grows in number of components and servers, extracting and analyzing relevant information from system and application logs becomes a complex task. This contribution focuses in an approach adopted by Port d'Informació Científica (PIC) to make sense out of billions of lines of data stored at our computing center logs. An indexed centralized storage has been established, which is composed of three main components.
We are using Logstash as log collector; elasticsearch as indexing engine, and Kibana is used as visualization interface. This approach provides a good performance in terms of searching.
The system itself is distributed in storage and in search computation terms and, is scalable and provides high availability. The system described allows to efficiently search, graph, analyze and make sense of a mountain of logs. This contribution shows how to build and setup the system, and how it is used in daily operations to ease in log searching.
Printable Summary
As a distributed computing environment grows in number of components and servers, extracting and analyzing relevant information from system and application logs becomes a complex task. This contribution focuses in an approach adopted by Port d'Informació Científica (PIC) to make sense out of billions of lines of data stored at our computing center logs. An indexed centralized storage has been established, which is composed of three main components.
We are using Logstash as log collector; elasticsearch as indexing engine, and Kibana is used as visualization interface. This approach provides a good performance in terms of searching.
The system itself is distributed in storage and in search computation terms and, is scalable and provides high availability. The system described allows to efficiently search, graph, analyze and make sense of a mountain of logs. This contribution shows how to build and setup the system, and how it is used in daily operations to ease in log searching.
