Speaker
Impact
ActiveMQ is considered a de-facto standard for data transportation of gLite-related services. The performed tests constitute an important step in the procedure that aims at bringing the new implementation in production. This activity has highlighted the reliability of the new DGAS implementation. The throughput measurements showed that the number of jobs processed with the new implementation is about half of those with the legacy mechanism. However, considering that these results are related to a first prototype with non-optimized code, we think improvements are possible in order to reach a throughput similar to that measured on the current production DGAS system.
Description of the work
This poster describes the new DGAS architecture and the test activities carried out to verify the functionality of the new implementation.
The new architecture tightly resembles the old one. The major difference is that it is now possible to implement record producers and consumers as plug-ins on the relevant nodes. The ActiveMQ enabled sensors are implemented via a plug-in that generates the usage record and publishes it on a general purpose ActiveMQ broker. A plug-in consumer then retrieves the record from the broker and feeds it into the already existing DGAS server implementation (HLR), where it is then processed just as the records coming through the old legacy transport protocol.
In order to check the correct implementation and the robustness of the system, a dedicated testbed has been set up at INFN, composed as follows:
- 1 CE with ActiveMQ enabled sensors
- 2 WNs
- 1 HLR with the latest production version (with legacy transport prototocol)
- 1 HLR prototype with ActiveMQ as transport protocol
- 1 dedicated ActiveMQ Broker
In addition , to check the correct usage records flow through a broker outside INFN, a general purpose ActiveMQ Broker (already available at CERN) has been used.
The first test aimed at checking the communication between CE and HLR server. Subsequently, a complete accounting cycle, from the collection of the information on the CE to the correct storage in the HLR database, has been verified. Another set of tests, aimed at checking the data integrity between the two different transport methods, has been carried out. Since all the tests have been successful we can conclude that the ActiveMQ-enabled DGAS implementation is able to process all the accounting data treated by the legacy protocol implementation, without any loss of usage records. In addition the system has been stress tested to check its reliability and robustness with positive results. Finally throughput measurements, in terms of jobs processed per day, have been performed.
URL
http://www.to.infn.it/dgas/
Conclusions
This poster shows the new architecture of DGAS accounting system, using ActiveMQ as transport layer. In particular the performed tests and their results are described in order to demonstrate the robustness of the service. The prototype will be improved before being deployed in the production infrastructure. This new implementation will be adopted by all the Italian Grid production infrastructure and all the other international institutions that deploy the DGAS service on their sites.
The next step will be the adoption of ActiveMQ to publish accounting data from the national repository into the central EGI one.
Overview
DGAS is a distributed tool for the implementation of national Grid accounting infrastructures. It is responsible for collecting and storing the usage records, which are then displayed in the HLRmon web portal and through the EGI Accounting Portal. Sensors are installed on the Computing Element (CE) to collect accounting information and convert it into usage records. Data are then sent to the HLR, a repository that persistently stores usage records for future processing. ActiveMQ has been selected as the standard transport mechanism in many gLite-related tools thus the DGAS development team decided to adopt it for the next releases by implementing a pluggable architecture on sensors and server side. ActiveMQ-based producers and consumers have then been implemented as plugins that can be installed on top of the already existing sensors, ensuring backward compatibility where needed. Record producers and consumers will be available as plug-in for both legacy and OGF-UR usage records.
