Big data challenge and operational processing needs

Extracted from Chimot, J., Global mapping of atmospheric composition from space – Retrieving aerosol height and tropospheric NO2 from OMI, PhD book, Delft University of Technology (TU Delft), The Royal Netherlands Meteorological Institute (KNMI), July 2018.


State-of-the-art retrieval algorithms are facing a continuous increase of the amount of observations: 1 million spectra per day from OMI (Levelt et al., 2006), 20 million spectra per day from the future TROPOMI on-board Sentinel-5 Precursor mission (Veefkind et al., 2012). The next operational Sentinel-4, Sentinel-5 and the proposed greenhouse gas Sentinel-7 missions will add even more observations (Ingmann et al., 2012). Increasing amounts of satellite data lead to several challenges regarding adequate analysis and the possibility to deliver near-real time information. Since implementation of space missions are based on big investments, we have to develop novel techniques, beyond the traditional ones, to guarantee relevant air quality and climate analyses from current and future satellite missions.

The next generation of satellite retrieval algorithms needs to address the challenges of increasing data amounts, by taking advantage from developments in other research areas with high impacts on society, such as the artificial intelligence techniques and machine learning algorithms devoted to fast detection, tracking and classification (Teichman and Thrun, 2012; Held et al., 2016). These new techniques have to be applied to improve the computational speed of satellite retrievals, but with careful adaptations to keep ensuring data with high quality and relevant scien- tific analysis.