Retrospective: active messages: a mechanism for integrating computation and communication

lr lh . e impact of Active Messages on the many facets of parallel computing surprised even us. Surely this success had an element of articulating the right concept at the right time. Like most good ideas, in hindsight it was present in some form or another in many places, and it certainly had roots in the architectural discussion of the time. Simplicity, flexibility, high-performance, combined with a clear cost model, were key in the acceptance of the Active Messages approach in many arenas. The underlying idea was simple: each message names a handler at the destination and on message arrival the handler is executed with the message as an argument. The handlers must execute quickly and cannot block. Their sole purpose is to extract the message out of the network and incorporate it into the on-going computation. Active Messages provides flexibility since handlers can be customized to particular communication instances, or less generally, to communication protocols or programming models. Active Messages implementations achieve high-performance because they eliminate complicated buffer management and simplify deadlock considerations. extension for communication and be used as a compilation target for languages that allowed finegrain communication. We had a straightforward goal: we wanted to measure interesting programs on a large scale and were ready to compile all the way down to the network on large parallel machines to cut down the communication overhead. The challenge we faced was that the basic communication operations available needed to deal with a large number issues, including routing, address translation, protection, buffering, output buffer full, deadlock freedom, event notification and more. Those thousand instructions in the commercial message passing layers were doing quite a bit. Although many machines could initiate a transfer to or from the network in a couple instructions, it took extremely careful engineering to address all of these issues in only a few more. While working on Active Messages, the handler-based messages of the Jmachine and Monsoon were fresh in our minds, but our view was that the communication instruction set should operate with fixed storage resources. Active Messages grew out of our work of comThere are several reasons for the success piling implicitly parallel dataflow languages, in Active Messages enjoyed. Timing and technologiparticular Id90, to commercial large-scale parallel cal advance were certainly important factors. Havmachines. Such languages require extremely fineing completed the nCUBE implementation, we grained communication and therefore a communiwere in a unique position to appreciate and exploit cation infrastructure supporting very low overthe capabilities of the CM-5 as soon as it appeared. head messages is essential. In the 1990-91 time Getting the overhead down to 2 l.ts made Active frame, industry was building interesting machines Messages on a commercial machine competitive on the scale of a thousand processors, but the with the contemporary research machines, and it architecture community took surprisingly little was only a matter of downloading a small library. notice. Mostly these machines were used to demWithin a few months of release it was installed on onstrate that message passing costs were very CM-5s throughout the world. Later, it was incorpohigh, but would be much lower on proposed rated into commercial products by Thinking designs -just as soon as they got built. We did not Machines, it was implemented on the Paragon and want to wait. Moreover, in looking closely at the incorporated into the OSF release, and recently it commercial architectures, the hardware was was incorporated into the IBM SP product. AM clearly capable of far better communication perforbecame the basis for much of the high performance mance than what was delivered. Thus, Active Mescluster work and the recent standardization effort sages was born of expediency: we sought to define in the Virtual Interface Architecture. Without the a simple, flexible communication primitive that caliber of the initial implementation and the willwould match what large parallel machines actually ingness to distribute and support the code, Active did well. This would serve as an instruction set Messages would probably not have enjoyed the