Process programming: passing into a new phase

Our whole experience, as a community, has been with a single type of program and its associated lifecycles, hereafter called Product Programs and Product Programming. Because of its importance to us as a multi-billion dollar industry, as our chosen profession, and as an intellectual challenge, we draw many distinctions within this broad category. For the same reasons, the Eskimo has 27 different words for snow to distinguish what for them are important differences. We naturally use this “vast” experience base to organize our view of the world. When new situations arise, we categorize them according to this existing structure, occasionally extending the structure by drawing finer distinctions. This is a very successful strategy and is the process by which community knowledge is built. However, this strategy is unsuccessful and quite debilitating on those rare occasions when truly new phenomena are encountered, because by operating at the micro level it tends to hide macro distinctions through a series of micro adaptations. Consider the dilemma of some hypothetical Eskimos who first encountered water and attempted to fit it into their “snow structure.” For them it was just a soft version of this rather obscure and troublesome form of snow called “slush.” This accommodation of the existing theory hid the fact that although there were some similarities and all the same “stuff” was involved, that because that material was in a different phase (as used technically in physics) it had fundamentally different properties which necessitated fundamentally different ways of handling it and different uses to which it was put. I believe that we find ourselves in just such a situation with process programming. It's constituted out of the same primitive components as product programs, but its organizational composition places it into a different “phase” of the material which gives it fundamentally different properties and forces us to handle and use it quite differently. We do ourselves a great disservice by adopting the following two assumptions without careful critical assessment:Process Programs are similar to Product Programs; Process Programming is similar to Product Programming. I believe these widely held assumptions are false and misleading and hope to demonstrate so in the rest of this position paper. Our limited process experience lies at the two ends of the spectrum:Very low level — routine procedures which are easily embedded (procedurally) in an automated tool. Very high level — broad organizational lifecycle models which have been so structured and regularized that they can be codified into formal models. At both of these levels, Process Programs have been expressed either procedurally (in some standard programming language) or via some state transition formalism. The conceptual basis for both is to identify all the options potentially available at some point and the preconditions necessary for employing each. Many corporations, partly driven by government pressure, are making major investments in codifying their lifecycle models and in automating their routine low level processes. Yet, I know of no interesting Process Program (with one exception described below). If you look at any of them, all you see is a program much like any other. How could it be otherwise, since we use the same formalisms as we do for Product Programs and conceptualize them similarly? Furthermore, even if you broaden the scope to include process programs from other fields, such as CAD, the results are remarkably similar — existing Process Programs mirror Product Programs. In fact, CAD has focused much more on prototyping support than on process support. They facilitate getting immediate feedback from models, and the construction and modification of those models. They thereby allow people to explore the implications of various designs and do tradeoff analysis, but the process is occurring outside the system. It is not captured, represented, or directly supported. Rather, the support arises from better tools for processing the artifacts resulting from the process. The single exception to this lack of interesting Process Program examples is Make (and all its descendents) which constructs the necessary process to build a configuration. What makes this class of Process Programs interesting is that it devised a special purpose model for describing its domain (configurations), structural properties (dependencies), and methods (build processes). Users instantiate this configuration model, and like other fourth generation application generators, the processor is able to “compile” the instantiated model, in this case into a Process Program for constructing the configuration. The key to this success was the invention of an appropriate model which allowed uses to occur by instantiation and in which any such instantiation could be processed by a fixed (and relatively simple) program. To me, this is the heart of Process Programming. Instantiation is the key. You don't write Process Programs, you instantiate and particularize existing ones. Furthermore, although some Process Programs (like Make) can be fully automated, many are specifically intended to include people and be interactive. These need not, and in fact, should not be fully instantiated before they begin. Rather, as more information is gathered or difficulties encountered, additional instantiation can occur to take advantage of the new information or overcome the difficulty. Such dynamic instantiation and monitoring is just what Planning Languages were designed for, and they seem a natural place to start looking for foundations for Process Programming. However, I expect that most of the leverage will come from a series of special purpose processors for particular domains (like Make) with their own languages and forms of instantiation. It should also be noted that instantiation necessarily implies reuse. Instantiation provides the basis for reusing preexisting models. The real bugaboo for reuse has always been adaptation. It is ludicrous to believe that you can find exactly what you want in any library of components; the space is simply too large. Instantiation is a way of adapting a general model through particularization, and is more general than simple parameterization. Even so, I expect that we will severely strain existing notions of instantiation in pursuing Process Programming and that, therefore, much of the insight and leverage will come from special purpose mechanisms. If this picture of Process Programs and Process Programming is accurate, then we need to treat them as separate entities rather than trying to force them into product formalisms and lifecycles. Nothing could be further from our current product preoccupation with building artifacts that satisfy a predefined static specification than a process world which incrementally defines and constructs its artifact (process) through instantiation based reuse while it is being used (executed). Clearly Process Programs and Process Programming are a phase of software as distinct from Product Programs and Product Programming as water is from snow. We must, therefore, not restrict our formalisms, techniques, and approaches to the former to those that have worked on the latter.