Swoozy - An Innovative Design of a Distributed and Gesture-based Semantic Television System

In this article, we describe an innovative approach to an intelligent television system named Swoozy that enables viewers to discover extended information such as facts, images, shopping recommendations or video clips about the currently broadcasted TV program by using the power of technologies of the Semantic Web (Web 3.0). Via a gesture-based user interface viewers will get answers to questions they may ask themselves during a movie or TV report directly on their television. In most cases, these questions are related to the name and vita of the featured actor, the place where a scene was filmed, or purchasable books and items about the topic of the report the viewer is watching. Furthermore, a new interaction concept for TVs is proposed using semantic annotations called “Grabbables” that are displayed on top of the videos and that provide a semantic referencing between the videos’ content and an ontological representation to access Semantic Web Services. Keywords–interactive television system; Semantic Web Technologies; Web 3.0; video annotation; gesture-based interaction.

[1]  Alberto Del Bimbo,et al.  Video Annotation and Retrieval Using Ontologies and Rule Learning , 2010, IEEE MultiMedia.

[2]  José Juan Pazos-Arias,et al.  AVATAR: an improved solution for personalized TV based on semantic inference , 2006, IEEE Transactions on Consumer Electronics.

[3]  Mathias Lux,et al.  Retrieval of MPEG-7 based Semantic Descriptions , 2005 .

[4]  Petri Vuorimaa,et al.  Decoding of DVB digital television subtitles , 2002 .

[5]  Jeremy J. Carroll,et al.  Resource description framework (rdf) concepts and abstract syntax , 2003 .

[6]  Frank Weichert,et al.  Analysis of the Accuracy and Robustness of the Leap Motion Controller , 2013, Sensors.

[7]  Hao Su,et al.  Object Bank: A High-Level Image Representation for Scene Classification & Semantic Feature Sparsification , 2010, NIPS.

[8]  Simon Bergweiler.  Interactive Service Composition and Query , 2014, Towards the Internet of Services.

[9]  Jerry R. Hobbs,et al.  DAML-S: Semantic Markup for Web Services , 2001, SWWS.

[10]  Díbio Leandro Borges,et al.  Structure in Soccer Videos: Detecting and Classifying Highlights for Automatic Summarization , 2005, CIARP.

[11]  Steffen Staab,et al.  Semantic Annotation of Images and Videos for Multimedia Analysis , 2005, ESWC.

[12]  Daniel Sonntag,et al.  Design and Implementation of Combined Mobile and Touchscreen-based Multimodal Web 3.0 Interfaces , 2009, IC-AI.

[13]  Daniel Porta,et al.  Building Multimodal Dialog User Interfaces in the Context of the Internet of Services , 2014, Towards the Internet of Services.

[14]  Mike Dowman,et al.  Semantically Enhanced Television News through Web and Video Integration , 2005 .

[15]  David A. McAllester,et al.  Object Detection with Discriminatively Trained Part Based Models , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  Alfred Kobsa,et al.  Generic User Modeling Systems , 2001, User Modeling and User-Adapted Interaction.

[17]  Óscar Corcho,et al.  The landscape of multimedia ontologies in the last decade , 2011, Multimedia Tools and Applications.

[18]  Daria Loi.  Changing the TV Industry through User Experience Design , 2011, HCI.

[19]  Daniel Porta,et al.  Integrating a multitouch kiosk system with mobile devices and multimodal interaction , 2010, ITS '10.

[20]  Sanggil Kang,et al.  An ontology-based personalized target advertisement system on interactive TV , 2011, 2011 IEEE International Conference on Consumer Electronics (ICCE).

[21]  Lora Aroyo,et al.  NoTube: The television experience enhanced by online social and semantic data , 2011, 2011 IEEE International Conference on Consumer Electronics -Berlin (ICCE-Berlin).

[22]  Özgür Ulusoy,et al.  A Semi-Automatic Semantic Annotation Tool for Video Databases , 2002 .

[23]  Huajun Chen,et al.  The Semantic Web , 2011, Lecture Notes in Computer Science.

[24]  E. Prud hommeaux,et al.  SPARQL query language for RDF , 2011 .

[25]  Alfred Kobsa,et al.  User Models in Dialog Systems , 1989, Symbolic Computation.

[26]  Nigel Shadbolt,et al.  Resource Description Framework (RDF) , 2009 .

[27]  J. Domingue,et al.  Towards semantic TV services a hybrid Semantic Web Services approach , 2010 .

[28]  Fernando Pereira,et al.  MPEG-7 the generic multimedia content description standard, part 1 - Multimedia, IEEE , 2001 .

[29]  Son Lam Phung,et al.  Automatic Image Annotation for Semantic Image Retrieval , 2007, VISUAL.

[30]  Mathias Lux,et al.  Emir : Semantics in Multimedia Retrieval and Annotation , 2004 .

[31]  Günter Neumann,et al.  Recognizing Textual Entailment Using Sentence Similarity based on Dependency Tree Skeletons , 2007, ACL-PASCAL@ACL.

[32]  Sebastian Rudolph,et al.  Semantic Web: Grundlagen , 2008 .

[33]  Norbert Reithinger,et al.  A Unified Approach for Semantic-Based Multimodal Interaction , 2014, Towards the Internet of Services.

[34]  Norbert Reithinger,et al.  A look under the hood: design and development of the first SmartWeb system demonstrator , 2005, ICMI '05.