Developing an Open Data Portal for the ESA Climate Change Initiative

We introduce the rationale for, and architecture of, the European Space Agency Climate Change Initiative (CCI) Open Data Portal ( http://cci.esa.int/data/ ). The Open Data Portal hosts a set of richly diverse datasets – 13 “Essential Climate Variables” – from the CCI programme in a consistent and harmonised form and to provides a single point of access for the (>100 TB) data for broad dissemination to an international user community. These data have been produced by a range of different institutions and vary across both scientific and spatio-temporal characteristics. This heterogeneity of the data together with the range of services to be supported presented significant technical challenges. An iterative development methodology was key to tackling these challenges: the system developed exploits a workflow which takes data that conforms to the CCI data specification, ingests it into a managed archive and uses both manual and automatically generated metadata to support data discovery, browse, and delivery services. It utilises both Earth System Grid Federation (ESGF) data nodes and the Open Geospatial Consortium Catalogue Service for the Web (OGC-CSW) interface, serving data into both the ESGF and the Global Earth Observation System of Systems (GEOSS). A key part of the system is a new vocabulary server, populated with CCI specific terms and relationships which integrates OGC-CSW and ESGF search services together, developed as part of a dialogue between domain scientists and linked data specialists. These services have enabled the development of a unified user interface for graphical search and visualisation – the CCI Open Data Portal Web Presence.

[1]  Steven C. Hankin.  The Live Access Server and DODS: Web visualization and data fusion for distributed holdings , 2001 .

[2]  Bryan Lawrence,et al.  Understanding Climate Data Through Commentary Metadata: The CHARMe Project , 2013, TPDL Workshops.

[3]  Lorenzo Bigagli,et al.  OGC® Catalogue Services 3.0 - General Model , 2016 .

[4]  R Lowry,et al.  Information in environmental data grids , 2008, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[5]  Duane E. Waliser,et al.  Satellite Observations for CMIP5: The Genesis of Obs4MIPs , 2014 .

[6]  Maarten Plieger,et al.  Developing a Metadata Infrastructure to facilitate data driven science gateway and to provide Inspire/GEMINI compliance for CLIPC , 2016 .

[7]  James Gallagher,et al.  OPeNDAP: Accessing data in a distributed, heterogeneous environment , 2003, Data Sci. J..

[8]  Stefano Nativi,et al.  Earth Science Infrastructures Interoperability: The Brokering Approach , 2013, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[9]  Bryan Lawrence,et al.  Storing and manipulating environmental big data with JASMIN , 2013, 2013 IEEE International Conference on Big Data.

[10]  William E. Allcock,et al.  The Globus Striped GridFTP Framework and Server , 2005, ACM/IEEE SC 2005 Conference (SC'05).

[11]  Ben Domenico,et al.  Design and implementation of netCDF markup language (NcML) and its GML-based extension (NcML-GML) , 2005, Comput. Geosci..

[12]  Aijun Chen,et al.  GEOSS Component and Service Registry: Design, Implementation and Lessons Learned , 2012, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[13]  Roy Lowry,et al.  The NERC Vocabulary Server: Version 2.0 , 2012 .

[14]  Bryan N. Lawrence,et al.  MOLES3: Implementing an ISO standards driven data catalogue , 2015 .

[15]  A. Cazenave,et al.  The ESA Climate Change Initiative: Satellite Data Records for Essential Climate Variables , 2013 .

[16]  Eric Guilyardi,et al.  Towards improved and more routine Earth system model evaluation in CMIP , 2016 .

[17]  Thomas E. Fricker,et al.  A verification framework for interannual-to-decadal predictions experiments , 2012, Climate Dynamics.

[18]  Satoko Horiyama Miura.  Earth Observation data access interoperability implementation among space agencies , 2016, 2016 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[19]  Sarah Callaghan,et al.  Twenty Years of Data Management in the British Atmospheric Data Centre , 2015, Int. J. Digit. Curation.