A demonstration system which performs efficient bulk processing of large amounts of satellite data. Calvalus aims at supporting Earth Observation cal/val activities and algorithm development by providing fast access to mission-wide satellite data and a highly configurable processing environment. The Calvalus system utilises Apache Hadoop and the ESA BEAM development platform and achieves its enhanced performance from the massive parallelisation and data-local execution of processing tasks.
Source Brockmann Consult
The ESA Calvalus Study
Objective
ESA’s Earth Observation (EO) missions provide a unique dataset of observational data of our environment. Calibration of the measured signal and validation of the derived products is an extremely important task for efficient exploitation of EO data and the basis for reliable scientific conclusions. In spite of this importance, the cal/val work is often hindered by insufficient means to access data, time consuming work to identify suitable in-situ data matching the EO data, incompatible software and limited possibilities for rapid prototyping and testing of ideas. In view of the future fleet of satellites and the fast-growing amount of data produced, a very efficient technological backbone is required to maintain the ability of ensuring data quality and algorithm performance.
The announcement of opportunities EO Cal/Val and User Services is a technology study of the ESA LET-SME 2009 call, investigating into an existing leading edge technology (LET) for their applicability in the EO domain. Specifically, LET-SME is a spin-in instrument encouraging the participation of SMEs to ESA technology. The LET-SME focuses on early stage development of Leading Edge Technologies, i.e. the ones likely to become the reference technologies for the near future, and have good chances of being infused into ESA projects and missions.
In accordance with the study’s statement of work, Calvalus is a system that has been proposed to fully support the idea of LET-SME, thus with a strong focus on a selected LET which is described in this report.