App-Based Task Shortcuts for Virtual Assistants

Virtual assistants like Google Assistant and Siri often interface with external apps when they cannot directly perform a task. Currently, developers must manually expose the capabilities of their apps to virtual assistants, using App Actions on Android or Shortcuts on iOS. This paper presents savant, a system that automatically generates task shortcuts for virtual assistants by mapping user tasks to relevant UI screens in apps. For a given natural language task (e.g., “send money to Joe”), savant leverages text and semantic information contained within UIs to identify relevant screens, and intent modeling to parse and map entities (e.g., “Joe”) to required UI inputs. Therefore, savant allows virtual assistants to interface with apps and handle new tasks without requiring any developer effort. To evaluate savant, we performed a user study to identify common tasks users perform with virtual assistants. We then demonstrate that savant can find relevant app screens for those tasks and autocomplete the UI inputs.

[1]  Xiaofeng Wang,et al.  UIPicker: User-Input Privacy Identification in Mobile Applications , 2015, USENIX Security Symposium.

[2]  Krish Perumal,et al.  VASTA: a vision and language-assisted smartphone task automation system , 2019, IUI.

[3]  Alexandros Karatzoglou,et al.  Climbing the app wall: enabling mobile app discovery through context-aware recommendations , 2012, CIKM '12.

[4]  Amos Azaria,et al.  SUGILITE: Creating Multimodal Smartphone Automation by Demonstration , 2017, CHI.

[5]  Guanling Chen,et al.  AppJoy: personalized mobile application discovery , 2011, MobiSys '11.

[6]  Suman Nath,et al.  PUMA: programmable UI-automation for large-scale dynamic analysis of mobile apps , 2014, MobiSys.

[7]  Suman Nath,et al.  uLink: Enabling User-Defined Deep Linking to App Content , 2016, MobiSys.

[8]  Jin-Hyuk Hong,et al.  Understanding and prediction of mobile application usage for smart phones , 2012, UbiComp.

[9]  Jacques Klein,et al.  FlowDroid: precise context, flow, field, object-sensitive and lifecycle-aware taint analysis for Android apps , 2014, PLDI.

[10]  Michael S. Bernstein,et al.  Iris: A Conversational Agent for Complex Tasks , 2017, CHI.

[11]  Daniel Gatica-Perez,et al.  Smartphone usage in the wild: a large-scale analysis of applications and context , 2011, ICMI '11.

[12]  Xuanzhe Liu,et al.  Mash Droid: An Approach to Mobile-Oriented Dynamic Services Discovery and Composition by In-App Search , 2015, 2015 IEEE International Conference on Web Services.

[13]  Yun Ma,et al.  Roaming across the Castle Tunnels: an Empirical Study of Inter-App Navigation Behaviors of Android Users , 2017, ArXiv.

[14]  Gierad Laput,et al.  CommandSpace: modeling the relationships between tasks, descriptions and features , 2014, UIST.

[15]  Yuanchun Li,et al.  Humanoid: A Deep Learning-Based Approach to Automated Black-box Android App Testing , 2019, 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[16]  Fanglin Chen,et al.  MessageOnTap: A Suggestive Interface to Facilitate Messaging-related Tasks , 2019, CHI.

[17]  Hui Xiong,et al.  Prediction for Mobile Application Usage Patterns , 2012 .

[18]  Oriana Riva,et al.  Kite: Building Conversational Bots from Mobile Apps , 2018, MobiSys.

[19]  Xin Rong,et al.  Managing Uncertainty in Time Expressions for Virtual Assistants , 2017, CHI.

[20]  Jeffrey Nichols,et al.  Rico: A Mobile App Dataset for Building Data-Driven Design Applications , 2017, UIST.

[21]  Alessandra Gorla,et al.  Checking app behavior against app descriptions , 2014, ICSE.

[22]  Karen Church,et al.  An In-Situ Study of Mobile App & Mobile Search Interactions , 2015, CHI.

[23]  Ranjitha Kumar,et al.  ERICA: Interaction Mining Mobile Apps , 2016, UIST.

[24]  Xuanzhe Liu,et al.  Aladdin: Automating Release of Deep-Link APIs on Android , 2018, WWW.

[25]  Seungyeop Han,et al.  TaintDroid , 2010, OSDI.

[26]  Fadi Biadsy,et al.  JustSpeak: enabling universal voice control on Android , 2014, W4A.

[27]  Hui Xiong,et al.  Mobile app recommendations with security and privacy awareness , 2014, KDD.

[28]  Lei Cen,et al.  Personalized Mobile App Recommendation: Reconciling App Functionality and User Privacy Preference , 2015, WSDM.

[29]  Zhaohui Wu,et al.  Discovering different kinds of smartphone users through their application usage behaviors , 2016, UbiComp.

[30]  Benjamin R. Cowan,et al.  "What can i help you with?": infrequent users' experiences of intelligent personal assistants , 2017, MobileHCI.

[31]  Peng Wang,et al.  AsDroid: detecting stealthy behaviors in Android applications by user interface and program behavior contradiction , 2014, ICSE.

[32]  Thomas F. Liu,et al.  Learning Design Semantics for Mobile Apps , 2018, UIST.

[33]  Byung-Gon Chun,et al.  TaintDroid: An Information-Flow Tracking System for Realtime Privacy Monitoring on Smartphones , 2010, OSDI.

[34]  Ricardo Baeza-Yates,et al.  Predicting The Next App That You Are Going To Use , 2015, WSDM.

[35]  Qiang Xu,et al.  Identifying diverse usage behaviors of smartphone apps , 2011, IMC '11.

[36]  Johannes Schöning,et al.  Falling asleep with Angry Birds, Facebook and Kindle: a large scale study on mobile application usage , 2011, Mobile HCI.

[37]  Adam Fourney,et al.  Query-feature graphs: bridging user vocabulary and system functionality , 2011, UIST '11.

[38]  Toby Jia-Jun Li,et al.  PUMICE: A Multi-Modal Agent that Learns Concepts and Conditionals from Natural Language and Demonstrations , 2019, UIST.

[39]  Donghwi Kim,et al.  X-Droid: A Quick and Easy Android Prototyping Framework with a Single-App Illusion , 2019, UIST.

[40]  Doug Belshaw.  Workflow | Powerful automation made simple. , 2015 .

[41]  Joonwon Lee,et al.  Hooked on smartphones: an exploratory study on smartphone overuse among college students , 2014, CHI.

[42]  Kaigui Bian,et al.  Characterizing Smartphone Usage Patterns from Millions of Android Users , 2015, Internet Measurement Conference.

[43]  Denzil Ferreira,et al.  Understanding the Challenges of Mobile Phone Usage Data , 2015, MobileHCI.