Toward understanding heterogeneity in computing

Heterogeneity complicates the efficient use of multicomputer platforms, but does it enhance their performance? their cost effectiveness? How can one measure the power of a heterogeneous assemblage of computers (“cluster,” for short), both in absolute terms (how powerful is this cluster) and relative terms (which cluster is more powerful)? What makes one cluster more powerful than another? Is one better off with a cluster that has one super-fast computer and the rest of “average” speed or with a cluster all of whose computers are “moderately” fast? If you could replace just one computer in your cluster with a faster one, which computer would you choose: the fastest? the slowest? How does one even ask questions such as these in a rigorous, yet tractable manner? A framework is proposed, and some answers are derived, a few rather surprising. Three highlights: (1) If one can replace only one computer in a cluster by a faster one, it is provably (almost) always most advantageous to replace the fastest one. (2) If the computers in two clusters have the same mean speed, then, empirically, the cluster with the larger variance in speed is (almost) always the faster one. (3) Heterogeneity can actually lend power to a cluster!

[1]  Jens Mache,et al.  Ray Tracing on Cluster Computers , 2000, PDPTA.

[2]  Larry Carter,et al.  Bandwidth-centric allocation of independent tasks on heterogeneous platforms , 2002, Proceedings 16th International Parallel and Distributed Processing Symposium.

[3]  P. Altena,et al.  In search of clusters , 2007 .

[4]  Arnold L. Rosenberg,et al.  Efficient trigger-broadcasting in heterogeneous clusters , 2005, J. Parallel Distributed Comput..

[5]  Ami Marowka,et al.  The GRID: Blueprint for a New Computing Infrastructure , 2000, Parallel Distributed Comput. Pract..

[6]  Pierre-François Dutot.  Master-slave tasking on heterogeneous processors , 2003, Proceedings International Parallel and Distributed Processing Symposium.

[7]  Arnold L. Rosenberg,et al.  On “Exploiting” Node-Heterogeneous Clusters Optimally , 2007, Theory of Computing Systems.

[8]  Da-Wei Wang,et al.  Reduction optimization in heterogeneous cluster environments , 2000, Proceedings 14th International Parallel and Distributed Processing Symposium. IPDPS 2000.

[9]  David Abramson,et al.  A case for economy grid architecture for service oriented grid computing , 2001, Proceedings 15th International Parallel and Distributed Processing Symposium. IPDPS 2001.

[10]  David P. Anderson,et al.  SETI@home-massively distributed computing for SETI , 2001, Comput. Sci. Eng..

[11]  Arnold L. Rosenberg.  Needed: a theoretical basis for heterogeneous parallel computing , 1994 .

[12]  Pangfeng Liu,et al.  Broadcast scheduling optimization for heterogeneous cluster systems , 2000, SPAA '00.

[13]  Arnold L. Rosenberg,et al.  Toward a theory for scheduling dags in Internet-based computing , 2006, IEEE Transactions on Computers.

[14]  Franck Cappello,et al.  An algorithmic model for heterogeneous hyper-clusters: rationale and experience , 2005, Int. J. Found. Comput. Sci..

[15]  Dhabaleswar K. Panda,et al.  Designing communication strategies for heterogeneous parallel systems , 1998, Parallel Comput..

[16]  Yves Robert,et al.  The master-slave paradigm with heterogeneous processors , 2001, Proceedings 42nd IEEE Symposium on Foundations of Computer Science.

[17]  Dhabaleswar K. Panda,et al.  Efficient collective communication on heterogeneous networks of workstations , 1998, Proceedings. 1998 International Conference on Parallel Processing (Cat. No.98EX205).

[18]  Keith Marzullo,et al.  The computational Co-op: Gathering clusters into a metacomputer , 1999, Proceedings 13th International Parallel Processing Symposium and 10th Symposium on Parallel and Distributed Processing. IPPS/SPDP 1999.

[19]  Rajkumar Buyya,et al.  Article in Press Future Generation Computer Systems ( ) – Future Generation Computer Systems Cloud Computing and Emerging It Platforms: Vision, Hype, and Reality for Delivering Computing as the 5th Utility , 2022 .

[20]  David E. Culler,et al.  A case for NOW (networks of workstation) , 1995, PODC '95.

[21]  Yves Robert,et al.  Scheduling Divisible Loads with Return Messages on Heterogeneous Master-Worker Platforms , 2005, HiPC.

[22]  Arnold L. Rosenberg.  An algorithmic model for heterogeneous clusters: rationale and experience , 2004, 18th International Parallel and Distributed Processing Symposium, 2004. Proceedings..

[23]  Ian Foster,et al.  The Grid 2 - Blueprint for a New Computing Infrastructure, Second Edition , 1998, The Grid 2, 2nd Edition.

[24]  Arnold L. Rosenberg,et al.  Statistical predictors of computing power in heterogeneous clusters , 2010, 2010 IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum (IPDPSW).

[25]  Yves Robert,et al.  Scheduling Strategies for Master-Slave Tasking on Heterogeneous Processor Grids , 2002, PARA.

[26]  Larry Carter,et al.  Scheduling strategies for master-slave tasking on heterogeneous processor platforms , 2004, IEEE Transactions on Parallel and Distributed Systems.

[27]  Viktor K. Prasanna,et al.  Efficient collective communication in distributed heterogeneous systems , 2003, J. Parallel Distributed Comput..