Sep 16 – 19, 2013
Meliá Castilla Convention Centre, Madrid
Europe/Madrid timezone

Setting up a log centralization infrastructure

Sep 16, 2013, 9:00 AM
8h 30m
Meliá Castilla Convention Centre, Madrid

Meliá Castilla Convention Centre, Madrid

Speakers

Bruno Rodriguez (IFAE) Jose Flix (IFAE)

Description of Work

As a distributed computing environment grows in number of components and servers, extracting and analyzing relevant information from system and application logs becomes a complex task. This contribution focuses in an approach adopted by Port d'Informació Científica (PIC) to make sense out of billions of lines of data stored at our computing center logs. An indexed centralized storage has been established, which is composed of three main components.
We are using Logstash as log collector; elasticsearch as indexing engine, and Kibana is used as visualization interface. This approach provides a good performance in terms of searching.
The system itself is distributed in storage and in search computation terms and, is scalable and provides high availability. The system described allows to efficiently search, graph, analyze and make sense of a mountain of logs. This contribution shows how to build and setup the system, and how it is used in daily operations to ease in log searching.

Printable Summary

As a distributed computing environment grows in number of components and servers, extracting and analyzing relevant information from system and application logs becomes a complex task. This contribution focuses in an approach adopted by Port d'Informació Científica (PIC) to make sense out of billions of lines of data stored at our computing center logs. An indexed centralized storage has been established, which is composed of three main components.
We are using Logstash as log collector; elasticsearch as indexing engine, and Kibana is used as visualization interface. This approach provides a good performance in terms of searching.
The system itself is distributed in storage and in search computation terms and, is scalable and provides high availability. The system described allows to efficiently search, graph, analyze and make sense of a mountain of logs. This contribution shows how to build and setup the system, and how it is used in daily operations to ease in log searching.

Primary author

Bruno Rodriguez (IFAE)

Co-authors

Carles Acosta Silva (IFAE) Ms Elena Planes (Collaborator) Mrs Esther Accion (Collaborator) Mr Fernando Lopez (Collaborator) Mr Jordi Casals (Collaborator) Jose Flix (IFAE) Marc Caubet (Port d'Informació Científica) Mr Ricard Cruz (Collaborator) Ms Vanessa Acin (Collaborator)

Presentation materials

There are no materials yet.