Performance requirements of real-time continuous media computing: specification techniques and hardware/software implications

Today's typical multimedia computer consists of a fast CPU and multimedia peripherals such as attached sound and video support hardware. In such a machine, the low-level audio and video handling may be performed by these peripherals while any significant processing on these media requires the intimate involvement of the CPU, main memory system and the operating system. Given that these components are not typically designed to meet the real-time requirements of continuous media, there are a number of constraints on the nature of multimedia applications that can be executed. To get around these constraints, demanding multimedia applications may typically be offloaded to attached peripheral compute engines which are customized to execute the applications. This approach requires dedicating resources to specific applications and suffers from the problem that such applications are not flexible integrated with the main system. We posit that new software and hardware technologies are needed to truly integrate real-time multimedia processing capabilities within a multimedia computer. The design of these technologies must be based on a thorough understanding of the timing and real-time computing requirements of various types of continuous media. Formal mechanisms are needed to express and analyze these requirements in order for multimedia operating systems to efficiently and correctly schedule the use of computing and communication resources. Indeed such mechanisms may be used to guide the specification and design of hardware and software processing component. In this paper, we present a framework for specifying the timing properties and computational requirements of continuous media. We discuss the usefulness of this framework in the context of some common audio and video data formats. We also discuss the impact of this framework on the design of resource scheduling mechanisms. Finally we offer some insights on designing audio/video processing engines based on real-time requirements.