Speaker
Description
In many scientific disciplines, expensive equipment is shared nowadays. The users – scientists, request specific experiments from facilities that perform them on their behalf. The outcome of such an experiment is a dataset, which can be quite huge in many cases. Our introduced system provides an easy way to make data produced by such specialized devices available to the scientific community. It is used to manage the storage of experimental data between several tiers of physical data storage consisting of the experimental facilities where data are acquired, national or scientific domain data storage services, and computing facilities provided on both national and European levels.
The software is built on the top of the Onedata system. It supports the whole process, from storing produced data from the device, setting up all necessary Onedata options, publishing the datasets, and archiving in permanent storage. It implements varying policies of handling the data, e.g., expiration at the acquisition facility, archiving in multiple copies, and data publication after an embargo period. It can also export datasets to supported repositories or metadata to metadata catalogues. The demonstrated application automatically controls the whole data workflow according to the defined Data Management Plan, which is attached to the dataset in a YAML file.
We are going to cover in our demonstration:
- briefly set up and run Oneprovider,
- setup our application,
- create a test dataset with metadata,
- run the data workflow with several configuration possibilities,
- access the dataset through Onedata web interface and CLI Oneclient,
- presentation of processing CryoEM data in Scipion adapted to run with Onedata in container and Kubernetes.
| Topic | Data Spaces |
|---|
