A Web-based Evaluation Framework for Spatial Instruction-Giving Systems

We demonstrate a web-based environment for development and testing of different pedestrian route instruction-giving systems. The environment contains a City Model, a TTS interface, a game-world, and a user GUI including a simulated street-view. We describe the environment and components, the metrics that can be used for the evaluation of pedestrian route instruction-giving systems, and the shared challenge which is being organised using this environment.

[1]  R. Malaka,et al.  DEEP MAP: Challenging IT Research In The Framework Of A Tourist Information System , 2000 .

[2]  Kallirroi Georgila,et al.  User simulation for spoken dialogue systems: learning and evaluation , 2006, INTERSPEECH.

[3]  Gaetano Borriello,et al.  Landmark-based pedestrian navigation from collections of geotagged photos , 2008, MUM '08.

[4]  YoungSteve,et al.  A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies , 2006 .

[5]  Oliver Lemon,et al.  Learning Lexical Alignment Policies for Generating Referring Expressions for Spoken Dialogue Systems , 2009, ENLG.

[6]  Sabine Geldof,et al.  CORAL: using natural language generation for navigational assistance , 2003 .

[7]  Oliver Lemon,et al.  The GRUVE Challenge: Generating Routes under Uncertainty in Virtual Environments , 2011, ENLG.

[8]  Johanna D. Moore,et al.  Shared Task Proposal: Instruction Giving in Virtual Worlds , 2007 .

[9]  Panos Markopoulos,et al.  GentleGuide: An Exploration of Haptic Output for Indoors Pedestrian Guidance , 2003, Mobile HCI.

[10]  David R. Morse,et al.  AudioGPS: Spatial Audio Navigation with a Minimal Attention Interface , 2002, Personal and Ubiquitous Computing.

[11]  Sean J. Barbeau,et al.  Positional Accuracy of Assisted GPS Data from High-Sensitivity GPS-enabled Mobile Phones , 2011, Journal of Navigation.

[12]  Kai-Florian Richter,et al.  Simplest Instructions: Finding Easy-to-Describe Routes for Navigation , 2008, GIScience.

[13]  Steve J. Young,et al.  A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies , 2006, The Knowledge Engineering Review.

[14]  Jon Oberlander,et al.  Generating Instructions in Virtual Environments (GIVE):A Challenge and an Evaluation Testbed for NLG , 2007 .

[15]  Matt Jones,et al.  ONTRACK: Dynamically adapting music playback to support navigation , 2008, Personal and Ubiquitous Computing.

[16]  Ashweeni Kumar Beeharee,et al.  A natural wayfinding exploiting photos in pedestrian navigation systems , 2006, Mobile HCI.