Towards Medical Cyber-Physical Systems: Multimodal Augmented Reality for Doctors and Knowledge Discovery about Patients

In the medical domain, which becomes more and more digital, every improvement in efficiency and effectiveness really counts. Doctors must be able to retrieve data easily and provide their input in the most convenient way. With new technologies towards medical cyber-physical systems, such as networked head-mounted displays (HMDs) and eye trackers, new interaction opportunities arise. With our medical demo in the context of a cancer screening programme, we are combining active speech based input, passive/active eye tracker user input, and HMD output (all devices are on-body and hands-free) in a convenient way for both the patient and the doctor.

[1]  Wolfgang Wahlster,et al.  SmartKom: Symmetric Multimodality in an Adaptive and Reusable Dialogue Shell , 2003 .

[2]  Ashwani Kumar,et al.  Miamm — A Multimodal Dialogue System Using Haptics , 2005 .

[3]  F. Hall,et al.  The radiology report of the future. , 2009, Radiology.

[4]  Piotr Indyk,et al.  Approximate Nearest Neighbor: Towards Removing the Curse of Dimensionality , 2012, Theory Comput..

[5]  Sharon L. Oviatt,et al.  Ten myths of multimodal interaction , 1999, Commun. ACM.

[6]  David L Weiss,et al.  Structured reporting: patient care enhancement or productivity nightmare? , 2008, Radiology.

[7]  N. Navab,et al.  Advanced Display and Visualization Concepts for Image Guided Surgery , 2008, Journal of Display Technology.

[8]  Norbert Reithinger,et al.  An integration framework for a mobile multimodal dialogue system accessing the semantic web , 2005, INTERSPEECH.

[9]  K. Keller,et al.  Head Mounted Displays for Medical Use , 2008, Journal of Display Technology.

[10]  Piotr Indyk,et al.  Approximate nearest neighbors: towards removing the curse of dimensionality , 1998, STOC '98.

[11]  Paul Buitelaar,et al.  Pillars of Ontology Treatment in the Medical Domain , 2009, J. Cases Inf. Technol..

[12]  Daniel Sonntag,et al.  Digital pen in mammography patient forms , 2011, ICMI '11.

[13]  Daniel Sonntag,et al.  RadSpeech's mobile dialogue system for radiologists , 2012, IUI '12.

[14]  Edward A. Lee.  Cyber Physical Systems: Design Challenges , 2008, 2008 11th IEEE International Symposium on Object and Component-Oriented Real-Time Distributed Computing (ISORC).

[15]  J. V. Kuppevelt,et al.  Advances in natural multimodal dialogue systems , 2005 .

[16]  Jukka Häkkinen,et al.  Midwives experiences of using HMD in ultrasound scan , 2006, NordiCHI '06.