terradue-eng - Tumblr blog

[terradue-eng]

Cloud processing with Cryosat data and SARvatore tools

(This is a summary, see related post for more details)

End 2013, the European Space Agency presented new GEOWOW results leveraging Terradue’s Cloud Platform: porting a scalable processing workflow for the CryoSat Satellite data, based on the "SARvatore" processor, on a Cloud Sandbox environment. Launched on 8 April 2010, CryoSat2 contributes to ESA’s ice mission by informing scientists on oceans, coastal areas, inland water bodies and even land surfaces. Cryosat's radar altimeter accurately measures variations in the height of the ice, sea level and sea ice’s height above water to derive sea-ice thickness. Terradue's Cloud Platform delivers a powerful Catalogue service, implementing the OGC standard interface "OpenSearch Geo and Time extensions". To create the OpenSearch Catalogue based on Cryosat metadata, we developed a RDF toolkit which parses the Header description files (.HDR) from Cryosat data package, and feeds the integrated SARvatore processor with data. Once the processing application is properly configured and deployed,  users can execute a Cloud processing workflow through a simple web interface.

[terradue-eng]

Cloud processing with Cryosat data and SARvatore tools

End 2013, the European Space Agency engaged with new developments within the "GEOSS interoperability for Weather, Ocean and Water (GEOWOW)" project, leveraging Terradue's Cloud technology. The objective was to implement a scalable processing workflow over the data delivered by the CryoSat Satellite, an ESA mission that is monitoring the changes in the thickness of marine ice (http://www.esa.int/Our_Activities/Observing_the_Earth/CryoSat/Earth_s_changing_ice). The processing workflow is based on the "SAR Versatile Altimetric Toolkit for Ocean & Land Research and Exploitation (SARvatore)", designed and developed by the Altimetry Team at ESA-ESRIN. It was evolved by the GEOWOW team while integrated within the provided Developer Cloud Sandboxes service. To feed the SARvatore processor, we also enabled the Cloud infrastructure with a new Catalogue service, configured over a subset of the FBR (Full Bit Rate) data downlinked uncompressed from the satellite (SAR/SARin mode) and the L1b Cryosat data (https://earth.esa.int/web/guest/-/products-overview-6975).

The Developer Cloud Sandbox service supports researchers in the development, testing and validation of their scientific application. The service then also provides drivers to deploy the application over a Cloud Computing cluster, for running a full processing campaign on virtually any third party provider. Our Cloud platform is powered with OpenNebula.org, and we regularly contribute open source drivers to the project. The Platform also delivers a powerful Catalogue service, leveraging a new standard: the OGC OpenSearch Geo and Time extension, is opening a new era for federating earth data resources on the open web.

In order to set a Developer Cloud Sandbox with the SARvatore utilities, the developer user has to install the main software package provided by the Altimetry Team at ESA-ESRIN, and then exploit the Sandbox PaaS environment to configure properly the application and the workflow streaming script (see the Github repository https://github.com/Terradue/sb-SARvatore/).

To create the OpenSearch Catalogue based on Cryosat metadata, we developed a toolkit which parses the Header description files (.HDR) from Cryosat data package (see an example of these data). The code of the conversion tools is planned for release on GitHub. Once the processing application is properly configured and deployed, end users are able to execute your CryoSat workflow through a web interface (provided by the Sandbox) as illustrated in this example:

Read more:

The GEOWOW project, vision and deliverables

[PDF] A Toolkit for CryoSat Investigations by the ESA-ESRIN EOP-SER R&D Altimetry Team, S. Dinardo (SERCO/ESRIN), B. Lucas (Deimos/ESRIN), J. Benveniste (ESA-ESRIN)

[terradue-eng]

Cloud Bursting: Exploiting the jclouds4one Add-on for OpenNebula

On Nov 15th, 2013 Terradue announced the new jclouds4one OpenNebula add-on. Co-funded by the European Space Agency, it is released in the frame of Terradue's "OpenNebula Add-ons" campaign.

This jclouds4one driver implements new capacities for hybrid Cloud Computing, expanding OpenNebula's support for Cloud Bursting, with the ability to work with a variety of up to 30 cloud providers & cloud software stacks, including Amazon, Azure, GoGrid, Ninefold, OpenStack, Rackspace, and vCloud.

The driver requires a command line interface, built on top of jclouds, like the jclouds-cli available on GitHub (https://github.com/jclouds/jclouds-cli). The driver interacts from one side with the OpenNebula Core, and from the other side with the jclouds CLI, that behaves as an interface upon a series of Cloud Providers.

To use the driver from your OpenNebula platform, you simply need the following:

Have a working account for a provider included in the jclouds library, and signup for a compute service (like EC2 in the case of AWS)

Download and unpack the jclouds-cli command line client software, and verify everything works with your credentials

Plug it and configure it into the OpenNebula front-end, by following an RPM-based installation or by a simple manual installation (see Documentation)

Create an OpenNebula template, inserting the relevant parameters like location, group, hardware type and so on (see Documentation)

Here is an example of a jclouds hybrid template:

NAME="jclouds" CONTEXT=[ FILES="file1 file2" ] JCLOUDS=[ GROUP="default", HARDWAREID="t1.micro", LOCATIONID="us-east-1d" ]

The CONTEXT keyword triggers the creation of a contextualization tar file, that enables a remote contextualization (i.e. the contextualization is made available on the main Cloud Controller and served via an HTTP server like Apache).

The following figure shows the jclouds4one driver in action, using the AWS EC2 provider through the jclouds library. The deployment ID gives information about the instance identifier and the public address assigned. Other information such as the private address are available under the tab Template.

Read more:

jclouds4one within the OpenNebula Add-Ons catalog

jclouds4one GitHub repository

jclouds4one Documentation