Speaker
Dr
Frailis Marco
(INAF - OATs)
Description
The EUCLID project is a medium-class mission of the ESA Cosmic Vision program. Its main objective is to map the geometry of the dark universe.
Euclid will generate 26 PB of data for each full data release (including external data from ground observations) but reprocessing needs and simulations will increase it to almost 200 PB. To handle this volume of data, the Euclid Science Ground Segment (SGS) federates 9 Science Data Centres (SDCs) and a Science Operations Centre, providing redundant and distributed data storage and processing.
To manage the heterogeneous computing and storage infrastructures of the SDCs, the SGS reference architecture is based on loosely coupled systems and services: 1) the Euclid Archive System (EAS), a central metadata repository which inventories, indexes and localizes the huge amount of distributed data; 2) a Distributed Storage System (DDS), providing a unified view of the SDCs storage systems and supporting several transfer protocols; 3) a COmmon ORchestration System (COORS), performing a balanced distribution of data and processing among the SDCs and executing processing plans based on user defined triggering criteria and Processing Functions (PFs); 4) an Infrastructure Abstraction Layer (IAL), isolating the processing software from the underlying IT infrastructure and providing a common, lightweight workflow management system; 5) a Monitoring & Control Service allowing to monitor the status of the SGS computing infrastructure as a whole or at SDC level; 6) a Common Data Model (CDM), a central repository where all SGS components interfaces and data structures are formalized in the XSD language.
Virtualization is another key element of the SGS infrastructure. The EuclidVM is a lightweight virtual machine, deployed in any SDC processing node, with a reference OS, selected stable software libraries and "dynamic" installation of the Euclid PFs.
These architecture concepts have been prototyped and are incrementally developed and validated through Euclid "SGS Challenges".
