Meeting with EISCAT-3D

Thursday 17 Oct 2013, 13:00 → 14:00 Europe/Amsterdam

Fourth meeting of EGI ENVRI Study Case with EISCAT_3D.

A study case was set up to identify existing services and solutions from EGI that could address the data pre-processing, post-processing, publishing needs of these two ESFRI projects. The outcome of the pilot is expected to be directly applicable to EISCAT_3D and EURO-ARGO, and indirectly by other ESFRIs of ENVRI. In cooperation with EISCAT_3D and EURO-ARGO representatives in ENVRI, EGI.eu will try to find best suitable solutions for data pre-processing of primary data and post-processing toward publishing.

More: https://wiki.egi.eu/wiki/EGI_ENVRI

Discussion 1
The discussion started with a brief analysis of H2020 work programme. Several related topics have been identified,  including but not limited to:
(1) Call1 continue development of ESFRI RIs and
(2) Call3 e-Infrastructures, in particular, topic 1 managing, preserving and computing with big research data, and topic 2 e-Infrastructure for Open Access. 
Some considerations:

1. Call1 PP2 starts from 2015 and EISCAT still needs to prepare for that. It is possible that we could target at Call3 first, then to prepare another proposal for call1 from later next year
2. Call 3 Topic 1 seems relevance, in particular items (6) which support to the evolution of EGI, and (7) proof of concept and prototypes of data infrastructure enabling software (EISFRI projects are encouraged) 
3. We need more analysis about the programme and wait for the final official calls.
4. Meantime, we shall make progress on the Pilot study, which will give good motivations in the proposal
5. Following on actions: see <Actions 2>

Ville asked whom should be included in the collaboration. Yin suggested that we may gradually identify and include other partners when possible. E.g., in the next to involve STFC.
Discussion 2
Then the discussion moved on to the pilot study. Gergely considered that we should give proper analysis on survey responses and user requirements. Some suggestions:

1. Interview those who answered the survey and to identify power users
2. Contact Ian McCrea and include STFC efforts
3. Study the EISCAT 3D working package deliverable 8.3, since it covers many work they’ve been done in understanding user requirements
4. Have a meeting with EISCAT 3D data working package to identify what design/implementation should be covered in the new investigation
5. Following on actions: see <Actions 1, 3>

The group suggested to start some practical experiments:

1. To move EISCAT 3D 1site 1year experimental data into EGI storage
2. To experiment some EGI and EUDAT services

EGI will contact EISCAT (Ingemar) about data formats and movements. Meeting shall be organised to discuss the details. Following on actions: see <Action 4>
Discussion 3
Yin suggested the following working packages could be planned in the proposal allowing us to work together:

1. Data acquisition: EISCAT 3D as the data provider to identify the urgent and important requirements which can be implemented in this proposal. (Joined efforts could include EISCAT, CU, CSC, STFC, Edinburgh …)
2. Data curation: EGI and EUDAT-centric, focuses on the supports to the evolution of EGI and integrations of EUDAT services. Gaps between generic service infrastructures and user requirements should be identified. Evolutions strategies and policy of usage should be made allowing user to adopt.  EISCAT 3D can be served as a use case to test the new (cloud-based or else) infrastructure services.  Some research can be explored here. (Joined efforts could include, EGI, EUDAT, EISCAT, CU, CSC, Amsterdam, CNRS …)
3. Data access and processing: research and proof-to-concept services/workflows for searching, analysing, and processing to unlock the value of petabyte datasets. (Joined efforts could include, CU, Edinburgh, Amsterdam, STFC, CNRS, EGI, EUDAT … )
4. Community support: an application store approach to train the new data scientists (Joined efforts could include, CU, CNRS, Edinburgh …) 

Discussion 4
Zhiming asked about how to catch up each other. Alex suggested to have monthly Skype discussion. Yin to set up Doodle for the next call (11 Nov -22 Nov)
<Actions>

1. Continue study / analyse survey responses
   
   * "Towards a Big Data Strategy for EISCAT-3D" document
   
   - on hold, future work need surveys analysis
   
   * MK to upload all the responses + Yin's document(s) to the DocDB page
   
   * All parties to analyses the responses from their perspective
   
   
   2) analysis Work programme for where to target
   
   - Longer term, strategic action.
   
   
   3) Cardiff University to set up a meeting with Ian / identify power users
   
   
   4) Call on accessing existing EISCAT archive from EGI
   
   - MK to setup a doodle, send it to EUDAT, Gergely, Ingemar and Mats Nylan. The doodle should cover Oct 23-25, Nov 4-8. The topic of discussion: Practical steps to make the 2TB size EISCAT dataset available on EGI/EUDAT