Roomy: a new approach to parallel disk-based computation

The design of a package, Roomy, for parallel disk-based computation is described. Many important algorithms run out of available RAM in minutes to hours. Roomy allows one to more easily produce CPU-intensive, storage-limited, data-parallel computations, while being minimally invasive with respect to the original data-parallel algorithm. Roomy supports a minimally invasive approach through its rich library of latency-tolerant parallel data structures that support delayed operations through a synchronization command. Roomy has been used to write some relatively short programs whose computations match those of some large, record-breaking computations reported in the recent literature.

[1]  Stuart Kauffman,et al.  Steps toward a possible theory of organization , 2004 .

[2]  Samuel A. Inverso,et al.  A genetic algorithm search for improved halftone masks , 2003 .

[3]  Gene Cooperman,et al.  Large implicit state space enumeration: overcoming memory and disk limitations , 2008 .

[4]  Gene Cooperman,et al.  DMTCP: Transparent checkpointing for cluster computations and the desktop , 2007, 2009 IEEE International Symposium on Parallel & Distributed Processing.

[5]  Nicholas Carriero,et al.  Linda and Friends , 1986, Computer.

[6]  Armin Biere,et al.  Bounded model checking , 2003, Adv. Comput..

[7]  Richard E. Korf.  Minimizing Disk I/O in Two-Bit Breadth-First Search , 2008, AAAI.

[8]  Nicholas Carriero,et al.  The S/Net's Linda kernel , 1986, TOCS.

[9]  Dragan Bosnacki,et al.  The Design of a Multicore Extension of the SPIN Model Checker , 2007, IEEE Transactions on Software Engineering.

[10]  Robert A. Whiteside,et al.  Implementing Linda for distributed and parallel processing , 1989, ICS '89.

[11]  D. Kunkle,et al.  Application of agent-based simulation to policy appraisal in the criminal justice system in England and Wales , 2004 .

[12]  Michael Isard,et al.  DryadLINQ: A System for General-Purpose Distributed Data-Parallel Computing Using a High-Level Language , 2008, OSDI.

[13]  S. VitterJ.,et al.  Algorithms for parallel memory, I , 1994 .

[14]  Gene Cooperman,et al.  Memory-based and disk-based algorithms for very high degree permutation groups , 2003, ISSAC '03.

[15]  Niklas Sörensson,et al.  An Extensible SAT-solver , 2003, SAT.

[16]  Jürgen Müller,et al.  A disk-based parallel implementation for direct condensation of large permutation modules , 2007, ISSAC '07.

[17]  Jeffrey Scott Vitter,et al.  Implementing I/O-efficient Data Structures Using TPIE , 2002, ESA.

[18]  Jiri Schindler,et al.  A load balancing framework for clustered storage systems , 2008, HiPC'08.

[19]  Richard D. Schlichting,et al.  Supporting Fault-Tolerant Parallel Programming in Linda , 1995, IEEE Trans. Parallel Distributed Syst..

[20]  Eric A. Hansen,et al.  Structured Duplicate Detection in External-Memory Graph Search , 2004, AAAI.

[21]  Richard E. Korf,et al.  Best-First Frontier Search with Delayed Duplicate Detection , 2004, AAAI.

[22]  Daniel Kunkle,et al.  Emergence of constraint in self-organizing systems. , 2004, Nonlinear dynamics, psychology, and life sciences.

[23]  Gene Cooperman,et al.  Twenty-six moves suffice for Rubik's cube , 2007, ISSAC '07.

[24]  Richard E. Korf,et al.  Large-Scale Parallel Breadth-First Search , 2005, AAAI.

[25]  Rob Pike,et al.  Interpreting the data: Parallel analysis with Sawzall , 2005, Sci. Program..

[26]  Moshe Y. Vardi,et al.  Memory-Ecient Algorithms for the Verication of Temporal Properties , 1992 .

[27]  Donald E. Knuth,et al.  The art of computer programming, volume 3: (2nd ed.) sorting and searching , 1998 .

[28]  Gene Cooperman,et al.  Solving Rubik's Cube: disk is the new RAM , 2008, CACM.

[29]  Assaf Schuster,et al.  A work-efficient distributed algorithm for reachability analysis , 2003, Formal Methods Syst. Des..

[30]  Xin Dong,et al.  Fast multiplication of large permutations for disk, flash memory and RAM , 2010, ISSAC.

[31]  Alan J. Hu,et al.  Protocol verification as a hardware design aid , 1992, Proceedings 1992 IEEE International Conference on Computer Design: VLSI in Computers & Processors.

[32]  Gerard J. Holzmann,et al.  Advanced SPIN Tutorial , 2004, SPIN.

[33]  Craig M. Chase,et al.  Distributed binary decision diagrams for verification of large circuits , 1996, Proceedings International Conference on Computer Design. VLSI in Computers and Processors.

[34]  Dragan Bosnacki,et al.  Multi-Core Model Checking with SPIN , 2007, 2007 IEEE International Parallel and Distributed Processing Symposium.

[35]  Joao Marques-Silva.  Practical applications of Boolean Satisfiability , 2008, 2008 9th International Workshop on Discrete Event Systems.

[36]  Alan J. Hu,et al.  BDDNOW: A Parallel BDD Package , 1998, FMCAD.

[37]  Jingren Zhou,et al.  SCOPE: easy and efficient parallel processing of massive data sets , 2008, Proc. VLDB Endow..

[38]  Gene Cooperman,et al.  A parallel architecture for disk-based computing over the Baby Monster and other large finite simple groups , 2006, ISSAC '06.

[39]  Yuan Yu,et al.  Dryad: distributed data-parallel programs from sequential building blocks , 2007, EuroSys '07.

[40]  Sanjay Ghemawat,et al.  MapReduce: Simplified Data Processing on Large Clusters , 2004, OSDI.

[41]  Assaf Schuster,et al.  Achieving Speedups in Distributed Symbolic Reachability Analysis Through Asynchronous Computation , 2005, CHARME.

[42]  Gene Cooperman,et al.  Parallel disk-based computation for large, monolithic binary decision diagrams , 2010, PASCO.

[43]  Wolfgang Rosenstiel,et al.  Distributed Symbolic Bounded Property Checking , 2006, PDMC@ICALP.

[44]  Gene Cooperman,et al.  Biased tadpoles: a fast algorithm for centralizers in large matrix groups , 2009, ISSAC '09.

[45]  Richard E. Korf,et al.  Recent Progress in Heuristic Search: A Case Study of the Four-Peg Towers of Hanoi Problem , 2007, IJCAI.

[46]  Gene Cooperman,et al.  Harnessing parallel disks to solve Rubik's cube , 2009, J. Symb. Comput..

[47]  Jeffrey Scott Vitter,et al.  Algorithms for parallel memory, I: Two-level memories , 2005, Algorithmica.

[48]  Peter J. Denning.  Virtual Memory , 1996, ACM Comput. Surv..

[49]  Gene Cooperman,et al.  A comparative analysis of parallel disk-based Methods for enumerating implicit graphs , 2007, PASCO '07.

[50]  GhemawatSanjay,et al.  The Google file system , 2003 .

[51]  Gianpiero Cabodi,et al.  Improved reachability analysis of large finite state machines , 1996, Proceedings of International Conference on Computer Aided Design.

[52]  Pierre Wolper,et al.  Simple on-the-fly automatic verification of linear temporal logic , 1995, PSTV.

[53]  Wilson C. Hsieh,et al.  Bigtable: A Distributed Storage System for Structured Data , 2006, TOCS.

[54]  Vipin Kumar,et al.  Parallel depth first search. Part I. Implementation , 1987, International Journal of Parallel Programming.

[55]  Forrest Brewer,et al.  Implementation of an efficient parallel BDD package , 1996, DAC '96.

[56]  Assaf Schuster,et al.  A Work-Efficient Distributed Algorithm for Reachability Analysis , 2003, CAV.

[57]  Enrico Tronci,et al.  Disk Based Software Verification via Bounded Model Checking , 2007, 14th Asia-Pacific Software Engineering Conference (APSEC'07).

[58]  Pierre Wolper,et al.  Memory-efficient algorithms for the verification of temporal properties , 1990, Formal Methods Syst. Des..

[59]  Alex Groce,et al.  New Challenges in Model Checking , 2008, 25 Years of Model Checking.

[60]  Pierre Wolper,et al.  An Automata-Theoretic Approach to Automatic Program Verification (Preliminary Report) , 1986, LICS.

[61]  Jeffrey Scott Vitter,et al.  External memory algorithms and data structures: dealing with massive data , 2001, CSUR.

[62]  Shin-ichi Minato,et al.  Streaming BDD Manipulation , 2002, IEEE Trans. Computers.

[63]  Donghui Zhang,et al.  Efficient mining of max frequent patterns in a generalized environment , 2006, CIKM '06.

[64]  D. E. Vengro.  A transparent parallel I/O environment , 1994 .

[65]  Peter Sanders,et al.  STXXL: standard template library for XXL data sets , 2008, Softw. Pract. Exp..