Trace2TAP: Synthesizing Trigger-Action Programs from Traces of Behavior

Two common approaches for automating IoT smart spaces are having users write rules using trigger-action programming (TAP) or training machine learning models based on observed actions. In this paper, we unite these approaches. We introduce and evaluate Trace2TAP, a novel method for automatically synthesizing TAP rules from traces (time-stamped logs of sensor readings and manual actuations of devices). We present a novel algorithm that uses symbolic reasoning and SAT-solving to synthesize TAP rules from traces. Compared to prior approaches, our algorithm synthesizes generalizable rules more comprehensively and fully handles nuances like out-of-order events. Trace2TAP also iteratively proposes modified TAP rules when users manually revert automations. We implemented our approach on Samsung SmartThings. Through formative deployments in ten offices, we developed a clustering/ranking system and visualization interface to intelligibly present the synthesized rules to users. We evaluated Trace2TAP through a field study in seven additional offices. Participants frequently selected rules ranked highly by our clustering/ranking system. Participants varied in their automation priorities, and they sometimes chose rules that would seem less desirable by traditional metrics like precision and recall. Trace2TAP supports these differing priorities by comprehensively synthesizing TAP rules and bringing humans into the loop during automation.

[1]  Qi Alfred Chen,et al.  ContexloT: Towards Providing Contextual Integrity to Appified IoT Platforms , 2017, NDSS.

[2]  Donatella Sciuto,et al.  BuildingRules: a trigger-action based system to manage complex commercial buildings , 2015, UbiComp/ISWC Adjunct.

[3]  Zhaohui Wu,et al.  Discovering different kinds of smartphone users through their application usage behaviors , 2016, UbiComp.

[4]  Maureen Schmitter-Edgecombe,et al.  Automated Cognitive Health Assessment From Smart Home-Based Behavior Data , 2016, IEEE Journal of Biomedical and Health Informatics.

[5]  Mark W. Newman,et al.  Devices and Data and Agents, Oh My , 2017, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[6]  Jiyun Lee,et al.  Trigger-Action Programming in the Wild: An Analysis of 200,000 IFTTT Recipes , 2016, CHI.

[7]  Fabio Paternò,et al.  Personalization of Context-Dependent Applications Through Trigger-Action Rules , 2017, ACM Trans. Comput. Hum. Interact..

[8]  Jo Vermeulen,et al.  From today's augmented houses to tomorrow's smart homes: new directions for home automation research , 2014, UbiComp.

[9]  Nick Feamster,et al.  Discovering Smart Home Internet of Things Privacy Norms Using Contextual Integrity , 2018, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[10]  Zhao Li,et al.  Systematically Debugging IoT Control System Correctness for Building Automation , 2016, BuildSys@SenSys.

[11]  Sumit Gulwani,et al.  From program verification to program synthesis , 2010, POPL '10.

[12]  Maya Cakmak,et al.  Supporting mental model accuracy in trigger-action programming , 2015, UbiComp.

[13]  Blase Ur,et al.  Rethinking Access Control and Authentication for the Home Internet of Things (IoT) , 2018, USENIX Security Symposium.

[14]  Qing Yang,et al.  Context-aware frame rate adaption for video chat on smartphones , 2013, UbiComp.

[15]  Volker Wulf,et al.  Evolving Needs in IoT Control and Accountability , 2018, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[16]  Blase Ur,et al.  How Users Interpret Bugs in Trigger-Action Programming , 2019, CHI.

[17]  Janardhan Rao Doppa,et al.  Learning Activity Predictors from Sensor Data: Algorithms, Evaluation, and Applications , 2017, IEEE Transactions on Knowledge and Data Engineering.

[18]  Ying Zhang,et al.  An empirical characterization of IFTTT: ecosystem, usage, and performance , 2017, Internet Measurement Conference.

[19]  Anind K. Dey,et al.  Support for context-aware intelligibility and control , 2009, CHI.

[20]  Carl A. Gunter,et al.  Charting the Attack Surface of Trigger-Action IoT Platforms , 2019, CCS.

[21]  Youn-Kyung Lim,et al.  User experience in do-it-yourself-style smart homes , 2015, UbiComp.

[22]  W. Keith Edwards,et al.  At Home with Ubiquitous Computing: Seven Challenges , 2001, UbiComp.

[23]  Sumit Gulwani,et al.  Dimensions in program synthesis , 2010, Formal Methods in Computer Aided Design.

[24]  Robert P. Nix,et al.  Editing by example , 1984 .

[25]  David Kim,et al.  Integrating the Smart Home into the Digital Calendar , 2016, CHI.

[26]  Franziska Roesner,et al.  Analysis of the Susceptibility of Smart Home Programming Interfaces to End User Error , 2019, 2019 IEEE Security and Privacy Workshops (SPW).

[27]  Prafulla N. Dawadi,et al.  Automated Clinical Assessment from Smart home-based Behavior Data , 2015 .

[28]  Anind K. Dey,et al.  Understanding and Using Context , 2001, Personal and Ubiquitous Computing.

[29]  P. Rousseeuw Silhouettes: a graphical aid to the interpretation and validation of cluster analysis , 1987 .

[30]  Joshua Zhexue Huang,et al.  Extensions to the k-Means Algorithm for Clustering Large Data Sets with Categorical Values , 1998, Data Mining and Knowledge Discovery.

[31]  SagivMooly,et al.  A simple inductive synthesis methodology and its applications , 2010 .

[32]  Fulvio Corno,et al.  Empowering End Users in Debugging Trigger-Action Rules , 2019, CHI.

[33]  Michael D. Ernst,et al.  Automatic Trigger Generation for Rule-based Smart Homes , 2016, PLAS@CCS.

[34]  Shan Lu,et al.  AutoTap: Synthesizing and Repairing Trigger-Action Programs Using LTL Properties , 2019, 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE).

[35]  Dongmei Zhang,et al.  Systematically Ensuring the Confidence of Real-Time Home Automation IoT Systems , 2018, ACM Trans. Cyber Phys. Syst..

[36]  Jun Gu,et al.  Algorithms for the satisfiability (SAT) problem: A survey , 1996, Satisfiability Problem: Theory and Applications.

[37]  Mark W. Newman,et al.  Learning from a learning thermostat: lessons for intelligent systems for the home , 2013, UbiComp.

[38]  Michael Weber,et al.  Exploring End User Programming Needs in Home Automation , 2017, ACM Trans. Comput. Hum. Interact..

[39]  Nikolaj Bjørner,et al.  Z3: An Efficient SMT Solver , 2008, TACAS.

[40]  Pamela Zave,et al.  Locked or Not?: Mental Models of IoT Feature Interaction , 2017, CHI.

[41]  Fabio Paternò,et al.  Supporting end-user debugging of trigger-action rules for IoT applications , 2019, Int. J. Hum. Comput. Stud..

[42]  Tom Rodden,et al.  "Playing with the Bits" User-Configuration of Ubiquitous Domestic Environments , 2003, UbiComp.

[43]  Mamun Bin Ibne Reaz,et al.  A Review of Smart Homes—Past, Present, and Future , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[44]  Diane J. Cook,et al.  Data-Driven Activity Prediction: Algorithms, Evaluation Methodology, and Applications , 2015, KDD.

[45]  Diane J. Cook,et al.  Keeping the Resident in the Loop: Adapting the Smart Home to the User , 2009, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[46]  Anind K. Dey,et al.  Is Context-Aware Computing Taking Control away from the User? Three Levels of Interactivity Examined , 2003, UbiComp.

[47]  Anind K. Dey,et al.  Assessing demand for intelligibility in context-aware applications , 2009, UbiComp.

[48]  Blase Ur,et al.  When Smart Devices Are Stupid: Negative Experiences Using Home Smart Devices , 2019, 2019 IEEE Security and Privacy Workshops (SPW).

[49]  Yung-Ju Chang,et al.  Exploring the Design Space of User-System Communication for Smart-home Routine Assistants , 2020, CHI.

[50]  Neil Immerman,et al.  A simple inductive synthesis methodology and its applications , 2010, OOPSLA.

[51]  Diane J. Cook,et al.  Discovering frequent user--environment interactions in intelligent environments , 2011, Personal and Ubiquitous Computing.

[52]  Luigi De Russis,et al.  A Debugging Approach for Trigger-Action Programming , 2018, CHI Extended Abstracts.

[53]  Sumit Gulwani,et al.  Oracle-guided component-based program synthesis , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[54]  Kyunghan Lee,et al.  Context-aware application scheduling in mobile systems: what will users do and not do next? , 2016, UbiComp.

[55]  Nikolaj Bjørner,et al.  Satisfiability modulo theories , 2011, Commun. ACM.

[56]  Lujo Bauer,et al.  Some Recipes Can Do More Than Spoil Your Appetite: Analyzing the Security and Privacy Risks of IFTTT Recipes , 2017, WWW.

[57]  Blase Ur,et al.  Practical trigger-action programming in the smart home , 2014, CHI.