Reinforcement learning for dynamic multimedia adaptation

In this paper we present an integration of several user and resource-related factors for the design of dynamic adaptation techniques. Our first contribution is an original reinforcement-learning approach to develop better adaptation agents. Integrated with the content, these agents improve gradually, by taking into account both user's behaviour and the usage context. Our second contribution is to apply this generic approach to solve an ubiquitous streaming problem. Mobile users experience large latencies while accessing streaming media. We propose to adapt the streaming by prefetching and to model this decision problem by using a Markov decision process. We discuss this formal framework and make explicit reference to its relationship with reinforcement learning. We support the benefits of our approach by presenting results from simulations and experiments.

[1]  R. Bellman Dynamic programming. , 1957, Science.

[2]  Mostafa H. Ammar,et al.  A comparison of layering and stream replication video multicast schemes , 2001, NOSSDAV '01.

[3]  Antonio F. Gómez-Skarmeta,et al.  Providing QoS through machine-learning-driven adaptive multimedia applications , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[4]  Stephen R. Gulliver,et al.  Pervasive and standalone computing: the perceptual effects of variable multimedia quality , 2004, Int. J. Hum. Comput. Stud..

[5]  Thomas Kunz Mobile code middleware for mobile multimedia information access , 2002, MOCO.

[6]  Bernd Girod,et al.  Advances in channel-adaptive video streaming , 2002, Proceedings. International Conference on Image Processing.

[7]  Lionel Brunie,et al.  Semantic Based Prefetching in News-on-Demand Video Servers , 2004, Multimedia Tools and Applications.

[8]  Peter Brusilovsky,et al.  Adaptive Hypermedia , 2001, User Modeling and User-Adapted Interaction.

[9]  Deborah Estrin,et al.  Multimedia proxy caching mechanism for quality adaptive streaming applications in the Internet , 2000, Proceedings IEEE INFOCOM 2000. Conference on Computer Communications. Nineteenth Annual Joint Conference of the IEEE Computer and Communications Societies (Cat. No.00CH37064).

[10]  Kevin C. Almeroth,et al.  Proceedings of the 13th international workshop on Network and operating systems support for digital audio and video , 2003 .

[11]  Martin L. Puterman,et al.  Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994 .

[12]  Pme Paul De Bra,et al.  Multimedia adaptation using AHA , 2004 .

[13]  George C. Polyzos,et al.  Adaptation techniques for ubiquitous Internet multimedia , 2001, Wirel. Commun. Mob. Comput..

[14]  Matthijs Douze,et al.  Optimizing hypervideo navigation using a Markov decision process approach , 2002, MULTIMEDIA '02.

[15]  Valentin S. Kisimov,et al.  Video Acceptability and Frame Rate , 1995, IEEE Multim..

[16]  Tayeb Lemlouma,et al.  Encoding multimedia presentations for user preferences and limited environments , 2003, 2003 International Conference on Multimedia and Expo. ICME '03. Proceedings (Cat. No.03TH8698).

[17]  Tayeb Lemlouma,et al.  Media Resources Adaptation for Limited Devices , 2003, ELPUB.

[18]  George D. Magoulas,et al.  Quality of service for perceptual considerations: an integrated perspective , 2001, IEEE International Conference on Multimedia and Expo, 2001. ICME 2001..

[19]  Lionel Brunie,et al.  Heuristics for Optimizing Multi-Clip Queries in Video Databases , 2004, Multimedia Tools and Applications.

[20]  Qian Zhang,et al.  QoS-adaptive proxy caching for multimedia streaming over the Internet , 2003, IEEE Trans. Circuits Syst. Video Technol..

[21]  Javed I. Khan,et al.  Partial Prefetch for Faster Surfing in Composite Hypermedia , 2001, USITS.

[22]  Jaideep Srivastava,et al.  Experimental evaluation of loss perception in continuous media , 1999, Multimedia Systems.

[23]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[24]  László Böszörményi,et al.  The Life Cycle of Multimedia Metadata , 2005, IEEE Multim..

[25]  Ingrid Zukerman,et al.  # 2001 Kluwer Academic Publishers. Printed in the Netherlands. Predictive Statistical Models for User Modeling , 1999 .

[26]  Christian Timmerer,et al.  Interoperable Adaptive Multimedia Communication , 2005, IEEE Multim..

[27]  Peter Brusilovsky,et al.  From adaptive hypermedia to the adaptive web , 2002, CACM.

[28]  Antonio F. Gómez-Skarmeta,et al.  Seamless Multimedia Communications in Heterogeneous Mobile Access Networks , 2004, TERENA Networking Conference.

[29]  Songqing Chen,et al.  Segment-based proxy caching for Internet streaming media delivery , 2005, IEEE MultiMedia.

[30]  Hae-Kwang Kim,et al.  AUDIO ANALYSIS AND ADAPTATION WITHIN INTERACTIVE MPEG-21 FRAMEWORK , 2003 .

[31]  Regunathan Radhakrishnan,et al.  Video Summarization Using Mpeg-7 Motion Activity and Audio Descriptors , 2003 .

[32]  Frank H. P. Fitzek,et al.  A prefetching protocol for continuous media streaming in wireless environments , 2001, IEEE J. Sel. Areas Commun..

[33]  Donald F. Towsley,et al.  Proxy prefix caching for multimedia streams , 1999, IEEE INFOCOM '99. Conference on Computer Communications. Proceedings. Eighteenth Annual Joint Conference of the IEEE Computer and Communications Societies. The Future is Now (Cat. No.99CH36320).

[34]  Multimedia Content Adaptation in MPEG-21 , 2006, Encyclopedia of Multimedia.