9–11 Oct 2018
Lisbon
Europe/Lisbon timezone

Building interoperable systems for SeaDataNet community

9 Oct 2018, 17:15
15m
Auditorium B104 (Lisbon)

Auditorium B104

Lisbon

ISCTE, University of Lisbon
Presentation Area 1. Cross-Domain challenges / Data exchange across domains: researchers, technologist and policy makers perspectives Data Management Services

Speaker

Chris Ariyo (CSC)

Description

The SeaDataNet project (https://www.seadatanet.org/) offers a robust and state-of-the-art Pan-European infrastructure to harmonise metadata and data from marine data centers in Europe, and offers the technology to make these data accessible. In the existing SeaDataNet common data index service (http://seadatanet.maris2.nl/v_cdi_v3/search.asp), data is residing at the data centers and is offered on demand across the user requests. However, the process is quite slow and also it was not easy to evaluate the quality of data, as data sets are directly uploaded by the data centers. To overcome this problem, the SeaDataNet community has partnered with the EUDAT CDI in the SeaDataCloud project to move its data to the EUDAT cloud storage and offer data directly from the cloud. Moreover, the community wants to perform quality checks on the data residing in the cloud before making it available for users. To implement the new upgraded system the existing SeaDataNet systems and EUDAT services have to be interoperable. This abstract discusses the solutions chosen for making different existing systems interoperable and the new infrastructure developed for the SeaDataNet common data index service. REST APIs are chosen to enable interaction between EUDAT services and community’s existing systems. Defining REST interfaces facilitated the understanding of different systems and helped in realizing the seamless communication between different systems. B2STAGE REST APIs (https://www.eudat.eu/b2stage) are used for all the interactions between the systems, such as uploading data to EUDAT cloud storage, downloading data from EUDAT cloud storage, performing transformations on the data in the cloud, etc. Moreover, in order to perform additional actions on the data in the cloud, such as checking the quality of data, performing transformations on data and for analyzing the data sets the existing EUDAT services are extended with new components. The EUDAT B2HOST (https://www.eudat.eu/services/userdoc/b2host) is extended to provide a container cluster that supports automatic data management in the cloud. The container cluster is managed using the EUDAT B2STAGE service, which allows systems to automatically run different tools on top of the data by interacting with its API. The technologies used for realizing data management in cloud are are Docker containers, RANCHER container platform, RabbitMQ and Elasticsearch, Logstash, and Kibana (ELK stack). The ability to offer automated data management in cloud could be valuable for other research communities as well. Moreover, the technical solutions chosen for developing the SeaDataNet common data index system could be used as reference solutions for building interoperable systems across different communities.

Summary

We present a new service that is developed together with the SeaDataNet community. The new service integrates three different systems developed at different research organisations and making different systems interoperable was a big challenge. We present the technical choices made for making the systems interoperable. Moreover, we also present a service for doing automatic data management in cloud. The choices made in developing these services could be used as reference solutions for building interoperable systems for new research communities.

Type of abstract Presentation

Primary author

Sri Harsha Vathsavayi (CSC)

Co-authors

Chris Ariyo (CSC) damien lecarpentier (CSC)

Presentation materials