Physical Relationship Description for Cyber-Physical Multi-modal Sensory Environments

This paper presents Physical Representation Description (PRD), a method for representing the spatial relationship between sources of multimedia streams in physical environments. PRD is an extension for the most widely used video streaming standard DASH. DASH supports the delivery of multiple synchronized streams (e.g., video, audio and sensor data) from a server to multiple clients, however DASH does not provide information about how the multi-modal sensory sources were actually located relative to each other when the data was captured. PRD extends DASH streaming standard by describing how the sources of multi-modal sensory streams are located relative to each other in the physical environment. This enables DASH clients to understand how the scene was captured in a physical space, and use this information to (1) differentiate and prioritize sensory streams according to their physical location, (2) enhance viewing experience, and (3) assist in easier spatial mapping from physical to virtual cyber space and vice versa. In this paper we describe the need for PRD in an example application called CyPhy, representing a multi-modal sensory cyberphysical environment for tele-physiotherapy in patients' homes. We then describe PRD and show how it can improve DASH streaming standard in delivering enhanced viewing experience under different network, and viewing conditions using physical spatial information of sensory sources.