A robust framework for real-time distributed processing of satellite data

It is estimated that future satellite instruments such as the Advanced Baseline Imager (ABI) and the Hyperspectral Environmental Suite (HES) on the GOES-R series of satellites will provide raw data volume of about 1.5Terabyte per day. Due to the high data rate, satellite ground data processing will require considerable computing power to process data in real-time. Cluster technologies employing a multi-processor system present the only current economically viable option. To sustain high levels of system reliability and operability in a cluster-oriented operational environment, a fault-tolerant data processing framework is proposed to provide a platform for encapsulating science algorithms for satellite data processing. The science algorithms together with the framework are hosted on a Linux cluster. In this paper we present an architectural model and a system prototype for providing performance, reliability, and scalability of candidate hardware and software for a satellite data processing system. Furthermore, benchmarking results are presented for a selected number of science algorithms for the Geosynchronous Imaging Fourier Transform Spectrometer (GIFTS) instrument showing that considerable performance can be gained without sacrificing the reliability and high availability constraints imposed on the operational cluster system.

[1]  James Arthur Kohl,et al.  Data redistribution and remote method invocation in parallel component architectures , 2005, 19th IEEE International Parallel and Distributed Processing Symposium.

[2]  David E. Bernholdt,et al.  Components, the Common Component Architecture, and the Climate/Weather/Ocean Community , 2003 .

[3]  Stephen D. Huston,et al.  The ACE Programmer's Guide: Practical Design Patterns for Network and Systems Programming , 2003 .

[4]  Kenneth A. Hawick,et al.  Distributed frameworks and parallel algorithms for processing large-scale geographic data , 2003, Parallel Comput..

[5]  B. Bouteiller,et al.  MPICH-V2: a Fault Tolerant MPI for Volatile Nodes based on Pessimistic Sender Based Message Logging , 2003, ACM/IEEE SC 2003 Conference (SC'03).

[6]  Robert O. Knuteson,et al.  GEOLOCATION OF THE GEOSYNCHRONOUS IMAGING FOURIER TRANSFORM SPECTROMETER (GIFTS) DATA , 2004 .

[7]  Jack Dongarra,et al.  MPI: The Complete Reference , 1996 .

[8]  Rob Gordon,et al.  Essential Jni: Java Native Interface , 1998 .

[9]  Craig J. Patten,et al.  DISCWorld: an environment for service-based matacomputing , 1999, Future Gener. Comput. Syst..

[10]  Joseph C. Coughlan,et al.  Distributed application framework for Earth science data processing , 2002, IEEE International Geoscience and Remote Sensing Symposium.

[11]  Hung-Lung Huang,et al.  Application of Principal Component Analysis to High-Resolution Infrared Measurement Compression and Retrieval , 2001 .

[12]  Raymond K. Garcia,et al.  Component-oriented design studies for efficient processing of hyperspectral infrared imager data , 2004, SPIE Optics + Photonics.

[13]  David R. O'Hallaron,et al.  Big Wins with Small Application-Aware Caches , 2004, Proceedings of the ACM/IEEE SC2004 Conference.

[14]  Keith Golden,et al.  Parallel Distributed Application Framework for Earth Science Data Processing , 2003, ScanGIS.

[15]  Anthony Skjellum,et al.  MPI/FT/sup TM/: architecture and taxonomies for fault-tolerant, message-passing middleware for performance-portable parallel computing , 2001, Proceedings First IEEE/ACM International Symposium on Cluster Computing and the Grid.