Speaker
Description
Wider impact and conclusions
This data platform will be used by different services that will be available on VERCE platform. Right now, this is used by VERCE's HPC use case, a forward modelling portal for storing and retrieving data and results which is generated from SuperMUC.
On successful completion of the prototype supporting both forward modelling and cross correlation use case, this setup can be deployed in other partner sites.
We are also considering a global catalog shared by all VERCE users and a query services to fetch the data in a better and simpler way.
Overall this helps seismologists in processing data and do simulations in a better way.
Description of work
Our test environment is setup in University of Edinburgh in Opennebula Virtual Machines which is federated with other institutes like CINECA, SCAI, IPGP, ISTerre and INGV
We have an iRODS installation to manage and federate users and sites preserving their policies and users. This iRODS installation provides a set of data management tools for users to upload download and view data and results in iRODS.
The iRODS catalog was relational and do not scale according to our requirement. So we had to use a distributed NoSQL meta-data catalog in MongoDB for data stored in iRODS. To keep the catalog up to date we created microservices to detect changes in data and update the catalog periodically
iRODS can support different file systems, they recently introduced support to HDFS. We have configured a hadoop cluster using virtual machines which can scale easily according to our needs and our current configuration gives user the option to store data in HDFS for archiving and distributed processing.
The data often has to be moved in and out of different HPC resources and the parallel data transfer provided by iRODS was not supported by these resources. With the help of a DSI module from CINECA, we were able to provide a GridFTP Interface for our iRODS installation.
URL(s) for further info
Verce Project
http://verce.eu
Forward modelling portal
http://129.215.213.249:8080/liferay-portal-6.1.0/
iRODS web interface
http://dir-irods.epcc.ed.ac.uk/irodsweb/