Jungle Computing: Distributed Supercomputing Beyond Clusters, Grids, and Clouds

In recent years, the application of high-performance and distributed computing in scientific practice has become increasingly wide spread. Among the most widely available platforms to scientists are clusters, grids, and cloud systems. Such infrastructures currently are undergoing revolutionary change due to the integration of many-core technologies, providing orders-of-magnitude speed improvements for selected compute kernels. With high-performance and distributed computing systems thus becoming more heterogeneous and hierarchical, programming complexity is vastly increased. Further complexities arise because urgent desire for scalability and issues including data distribution, software heterogeneity, and ad hoc hardware availability commonly force scientists into simultaneous use of multiple platforms (e.g., clusters, grids, and clouds used concurrently). A true computing jungle .

[1]  H. Bal,et al.  WebPIE : a Web-scale Parallel Inference Engine , 2010 .

[2]  Henri E. Bal,et al.  User-friendly and reliable grid computing based on imperfect middleware , 2007, Proceedings of the 2007 ACM/IEEE Conference on Supercomputing (SC '07).

[3]  Dennis Koelma,et al.  Finite state machine-based optimization of data parallel regular domain problems applied in low-level image processing , 2004, IEEE Transactions on Parallel and Distributed Systems.

[4]  Jason Maassen,et al.  Smartsockets: solving the connectivity problems in grid computing , 2007, HPDC '07.

[5]  The importance of technological advances , 2000, Nature Cell Biology.

[6]  A.J. Plaza,et al.  Recent developments and future directions in parallel processing of remotely sensed hyperspectral images , 2009, 2009 Proceedings of 6th International Symposium on Image and Signal Processing and Analysis.

[7]  Jack Dongarra,et al.  TOP500 Supercomputer sites 11/2000 - eScholarship , 2000 .

[8]  Ian J. Taylor,et al.  Distributed computing with Triana on the Grid , 2005, Concurr. Pract. Exp..

[9]  Frank van Harmelen,et al.  Semantic Web Technologies as the Foundation for the Information Infrastructure , 2008 .

[10]  Chein-I Chang,et al.  Hyperspectral Data Exploitation , 2007 .

[11]  David Abramson,et al.  Nimrod: a tool for performing parametrised simulations using distributed workstations , 1995, Proceedings of the Fourth IEEE International Symposium on High Performance Distributed Computing.

[12]  Jason Maassen,et al.  Real-World Distributed Computer with Ibis , 2010, Computer.

[13]  Randal A. Koene,et al.  NETMORPH: A Framework for the Stochastic Generation of Large Scale Neuronal Networks With Realistic Neuron Morphologies , 2009, Neuroinformatics.

[14]  Edward A. Lee,et al.  Scientific workflow management and the Kepler system , 2006, Concurr. Comput. Pract. Exp..

[15]  Dieter Fensel,et al.  Towards LarKC: A Platform for Web-Scale Reasoning , 2008, 2008 IEEE International Conference on Semantic Computing.

[16]  David Abramson,et al.  Bridging organizational network boundaries on the grid , 2005, The 6th IEEE/ACM International Workshop on Grid Computing, 2005..

[17]  Antonio Plaza,et al.  Parallel heterogeneous CBIR system for efficient hyperspectral image retrieval using spectral mixture analysis , 2010 .

[18]  Kees Verstoep,et al.  Wide-area communication for grids: an integrated solution to connectivity, performance and security problems , 2004, Proceedings. 13th IEEE International Symposium on High performance Distributed Computing, 2004..

[19]  Francisco Vilar Brasileiro,et al.  Faults in grids: why are they so bad and what can be done about it? , 2003, Proceedings. First Latin American Web Congress.

[20]  Jessica A. Faust,et al.  Imaging Spectroscopy and the Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) , 1998 .

[21]  Walter L. Warnick,et al.  The Digital Road to Scientific Knowledge Diffusion: A Faster, Better Way to Scientific Progress? , 2006, D Lib Mag..

[22]  Arnold W. M. Smeulders,et al.  A Minimum Cost Approach for Segmenting Networks of Lines , 2001, International Journal of Computer Vision.

[23]  Bruce G Buchanan,et al.  Automating Science , 2009, Science.

[24]  James Demmel,et al.  A view of the parallel computing landscape , 2009, CACM.

[25]  Antonio Plaza,et al.  Comparative analysis of different implementations of a parallel algorithm for automatic target detection and classification of hyperspectral images , 2009, Optical Engineering + Applications.

[26]  James A. Hendler,et al.  Web science: an interdisciplinary approach to understanding the web , 2008, CACM.

[27]  Sisi Zlatanova,et al.  Creating Spatial Information Infrastructures: Towards the Spatial Semantic Web , 2008 .

[28]  Gregory E. Chamitoff,et al.  Orders-of-magnitude performance increases in GPU-accelerated correlation of images from the International Space Station , 2010, Journal of Real-Time Image Processing.

[29]  Jason Maassen,et al.  JEL: unified resource tracking for parallel and distributed applications , 2011, Concurr. Comput. Pract. Exp..

[30]  Jason Maassen,et al.  Experiences with Fine-Grained Distributed Supercomputing on a 10G Testbed , 2008, 2008 Eighth IEEE International Symposium on Cluster Computing and the Grid (CCGRID).

[31]  Uwe Rascher,et al.  FLEX — Fluorescence Explorer: A Remote Sensing Approach to Quantify Spatio-Temporal Variations of Photosynthetic Efficiency from Space , 2008 .

[32]  T. Kielmann,et al.  Real-world Distributed Computing with Ibis , 2010 .

[33]  Ian T. Foster,et al.  The Anatomy of the Grid: Enabling Scalable Virtual Organizations , 2001, Int. J. High Perform. Comput. Appl..

[34]  Niels Drost,et al.  User Transparent Task Parallel Multimedia Content Analysis , 2010, Euro-Par.

[35]  Marcel Worring,et al.  The Semantic Pathfinder: Using an Authoring Metaphor for Generic Multimedia Indexing , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[36]  Gustavo Carneiro,et al.  Supervised Learning of Semantic Classes for Image Annotation and Retrieval , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[37]  Kathleen M. Carley.  Organizational Change and the Digital Economy: A Computational Organization Science Perspective , 2001 .

[38]  Adam Wierzbicki,et al.  Guest editors' introduction: Foundation of peer-to-peer computing , 2008, Comput. Commun..

[39]  Chein-I. Chang.  Hyperspectral Data Exploitation: Theory and Applications , 2007 .

[40]  A F Goetz,et al.  Imaging Spectrometry for Earth Remote Sensing , 1985, Science.

[41]  Jason Maassen,et al.  Self-adaptive applications on the grid , 2007, PPoPP.

[42]  Jason Maassen,et al.  Towards user transparent parallel multimedia computing on GPU-Clusters , 2010, ISCA'10.

[43]  Danny Crookes,et al.  Efficient implementation of a portable parallel programming model for image processing , 1999 .

[44]  Declan Butler.  The petaflop challenge , 2007, Nature.

[45]  Marcel Worring,et al.  High-Performance Distributed Image and Video Content Analysis with Parallel-Horus , 2007 .

[46]  Dennis Koelma,et al.  A software architecture for user transparent parallel image processing , 2002, Parallel Comput..

[47]  John F. Allen,et al.  Photosynthesis : energy from the sun : 14th International Congress on Photosynthesis , 2008 .

[48]  Henri E. Bal,et al.  Parallel simulation of ion recombination in nonpolar liquids , 1998, Future Gener. Comput. Syst..

[49]  Antonio J. Plaza,et al.  Commodity cluster-based parallel processing of hyperspectral imagery , 2006, J. Parallel Distributed Comput..

[50]  R. Douglas,et al.  Recurrent neuronal circuits in the neocortex , 2007, Current Biology.