Sorina POP (CNRS)
The DIRAC Project provides software for building large scale distributed computing systems . Services created using this software allow to integrate computing and storage resources of different types located anywhere in the world in a coherent federation with a single access point from the user perspective. The Project was initially started to operate grid computing resources for one of the experiments, LHCb, on the Large Hadron Collider (LHC) at CERN, Geneva, as a response to a challenge of processing unprecedented data volumes collected at LHC. The solution developed for LHCb was later adopted by other experiments in the High Energy Physics, Astrophysics and other scientific domains. Services based on the DIRAC software are also offered to the users of large national grid infrastructures, e.g. France-Grille National Grid Infrastructure (NGI) in France. Since 2014, the DIRAC services are also provided for the users of the European grid infrastructure (EGI). As a result, even small user communities can benefit from an easy access to vast amounts of resources available in these infrastructures. DIRAC services include both Workload and Data Management tools. The Workload Management Service (WMS) integrates resources provisioned by grid and cloud infrastructures but also by supercomputers, standalone computing farms or even volunteer computing systems. User tasks are submitted to the central Task Queue and then sent for execution to the resources matching the task requirements taking into account the task priorities and resource usage policies. Special attention is paid to automation of the management of large numbers of tasks that can be organized in complex workflows driven by the availability of data to be processed. Users access the service using different interfaces – command line tools, Web Portal, RESTful interface or programming API. The work pattern is similar to the one of using a large-scale batch system distributed worldwide. For example the DIRAC WMS of the LHCb experiment is capable of executing of up to 150K user tasks simultaneously, effectively operating a virtual computing center distributed over more than 120 sites over the world with a total capacity of 150K computing cores. The Data Management System (DMS) of DIRAC provides access to different kinds of data storage systems with virtually any access protocol existing in infrastructures supporting scientific research. The File Catalog service keeps track of all the physical copies of existing files and provides means for user defined metadata which allows efficient selection of datasets for a specific user analysis task. Support for massive data operations is also provided. In the whole, users perceive various computing and storage resources operated by the DIRAC services as a single large computer with a large distributed file system. The DIRAC software architecture is conceived for the use by various applications in different scientific domains. The modular structure of the software allows for easy extensions by developing additional components specific to workflows of particular user communities using the same software framework.
|Type of abstract||Poster|
Andrei Tsaregorodtsev (CNRS) Mr Fabio HERNANDEZ (CNRS) Mr Johan Bregeon (LUPM) Pierre Gay (CNRS) Sorina POP (CNRS) Vanessa Hamar (CNRS) arrabito arrabito (CNRS)