Stream processing

Stream processing is a basic method of code optimization related to loop fusion that can improve the space and speed of iterative applicative expressions by a process of loop combination. It has been studied before in applications to program improvement and batch oriented database restructuring. Previous attempts at implementation have been either highly restrictive, have required manual intervention, or have involved naive strategies resulting in impractically slow running times. This paper defines a new model for a stream processing problem for handling a wide class of applicative expressions that can be evaluated by looping in an unordered way through a single set or tuple valued argument. This problem is formulated graph theoretically and shown to be NP-Hard, even in the presence of constraints. Two new efficient heuristic algorithms will be presented. The efficiency of these solutions allows stream processing to be applied dynamically as a database query optimization, and as an important component of a high level language optimizer. Our method of stream processing has been implemented and used effectively within the RAPTS transformational programming system.

[1]  Daniel P. Friedman,et al.  Aspects of applicative programming for file systems (Preliminary Version) , 1977, Language Design for Reliable Software.

[2]  Robert Paige,et al.  Applications of Finite Differencing to Database Integrity Control and Query/Transaction Optimization , 1982, Advances in Data Base Theory.

[3]  Daniel P. Friedman,et al.  Aspects of applicative programming for file systems (Preliminary Version) , 1977 .

[4]  Richard C. Waters,et al.  Expressional loops , 1984, POPL.

[5]  Martin S. Feather,et al.  A system for developing programs by transformation , 1979 .

[6]  Robert Paige,et al.  Finite Differencing of Computable Expressions , 1982, TOPL.

[7]  Amir Pnueli,et al.  Use of a Nonprocedural Specification Language and Associated Program Generator in Software Development , 1979, TOPL.

[8]  J. Reif,et al.  Deriving Efficient Graph Algorithms* , 1984 .

[9]  J. D. Uiiman,et al.  Principles of Database Systems , 2004, PODS 2004.

[10]  Daniel P. Friedman,et al.  CONS Should Not Evaluate its Arguments , 1976, ICALP.

[11]  Robert Paige,et al.  Transformational programming: applications to algorithms and systems , 1983, POPL '83.

[12]  Barron C. Housel.  Pipelining: a technique for implementing data restructurers , 1979, TODS.

[13]  Mark H. Overmars,et al.  The Design of Dynamic Data Structures , 1987, Lecture Notes in Computer Science.

[14]  Leonidas J. Guibas,et al.  Compilation and delayed evaluation in APL , 1978, POPL.

[15]  Jacob T. Schwartz,et al.  On programming : an interim report on the SETL Project , 1973 .

[16]  Kang-Sen Lu.  Program optimization based on a non-procedural specification , 1981 .

[17]  John Darlington,et al.  A Transformation System for Developing Recursive Programs , 1977, J. ACM.

[18]  Patricia G. Selinger,et al.  Access path selection in a relational database management system , 1979, SIGMOD '79.

[19]  Micha Sharir.  Formal Integration: A Program Transformation Technique , 1981, Comput. Lang..

[20]  Ronald Fagin,et al.  Extendible hashing—a fast access method for dynamic files , 1979, ACM Trans. Database Syst..

[21]  John Randal Allen,et al.  Dependence analysis for subscripted variables and its application to program transformations , 1983 .

[22]  Paul B. Schneck,et al.  A survey of compiler optimization techniques , 1973, ACM Annual Conference.

[23]  Thomas I. M. Ho.  Review of On programming: an interim report on the SETL project, intallment II: the SETL language and examples of its use by J. T. Schwartz. New York University, 1973. , 1974, SIGP.