Programmatic and direct manipulation, together at last

Direct manipulation interfaces and programmatic systems have distinct and complementary strengths. The former provide intuitive, immediate visual feedback and enable rapid prototyping, whereas the latter enable complex, reusable abstractions. Unfortunately, existing systems typically force users into just one of these two interaction modes. We present a system called Sketch-n-Sketch that integrates programmatic and direct manipulation for the particular domain of Scalable Vector Graphics (SVG). In Sketch-n-Sketch, the user writes a program to generate an output SVG canvas. Then the user may directly manipulate the canvas while the system immediately infers a program update in order to match the changes to the output, a workflow we call live synchronization. To achieve this, we propose (i) a technique called trace-based program synthesis that takes program execution history into account in order to constrain the search space and (ii) heuristics for dealing with ambiguities. Based on our experience with examples spanning 2,000 lines of code and from the results of a preliminary user study, we believe that Sketch-n-Sketch provides a novel workflow that can augment traditional programming systems. Our approach may serve as the basis for live synchronization in other application domains, as well as a starting point for yet more ambitious ways of combining programmatic and direct manipulation.

[1]  Sumit Gulwani,et al.  Synthesizing geometry constructions , 2011, PLDI '11.

[2]  Anthony C. Davison,et al.  Bootstrap Methods and Their Application , 1998 .

[3]  Sumit Gulwani,et al.  From program verification to program synthesis , 2010, POPL '10.

[4]  Michel Beaudouin-Lafon,et al.  Instrumental interaction: an interaction model for designing post-WIMP user interfaces , 2000, CHI.

[5]  Zohar Manna,et al.  A Deductive Approach to Program Synthesis , 1979, TOPL.

[6]  Sumit Gulwani,et al.  FlashRelate: extracting relational data from semi-structured spreadsheets using examples , 2015, PLDI.

[7]  Armando Solar-Lezama,et al.  Program synthesis by sketching , 2008 .

[8]  Sumit Gulwani,et al.  User Interaction Models for Disambiguation in Programming by Example , 2015, UIST.

[9]  GulwaniSumit.  Automating string processing in spreadsheets using input-output examples , 2011 .

[10]  Andy Schürr,et al.  Dagstuhl seminar on bidirectional transformations (BX) , 2011, SGMD.

[11]  Sumit Gulwani,et al.  QuickDraw: improving drawing experience for geometric diagrams , 2012, CHI.

[12]  Emina Torlak,et al.  Angelic debugging , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[13]  Ravi Chugh,et al.  Prodirect Manipulation: Bidirectional Programming for the Masses , 2015, 2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C).

[14]  Sumit Gulwani,et al.  Automating string processing in spreadsheets using input-output examples , 2011, POPL '11.

[15]  Nikolaj Bjørner,et al.  Z3: An Efficient SMT Solver , 2008, TACAS.

[16]  Sumit Gulwani,et al.  Automated feedback generation for introductory programming assignments , 2012, PLDI.

[17]  Alan Borning,et al.  The Programming Language Aspects of ThingLab, a Constraint-Oriented Simulation Laboratory , 1981, TOPL.

[18]  Sumit Gulwani,et al.  Spreadsheet table transformations from examples , 2011, PLDI '11.

[19]  Umut A. Acar,et al.  Implicit self-adjusting computation for purely functional programs , 2011, J. Funct. Program..

[20]  Ruzica Piskac,et al.  Complete functional synthesis , 2010, PLDI '10.

[21]  Wilmot Li,et al.  Lillicon: using transient widgets to create scale variations of icons , 2015, ACM Trans. Graph..

[22]  Niklas Elmqvist,et al.  Direct manipulation through surrogate objects , 2011, CHI.

[23]  Viktor Kuncak,et al.  Synthesis modulo recursive functions , 2013, OOPSLA.

[24]  Benjamin C. Pierce,et al.  Boomerang: resourceful lenses for string data , 2008, POPL '08.

[25]  Sorin Lerner,et al.  Interactive parser synthesis by example , 2015, PLDI.

[26]  Anneli Folkesson,et al.  World Wide Web Consortium (W3C) , 2005 .

[27]  Marsha Chechik,et al.  Tools and Algorithms for the Construction and Analysis of Systems , 2016, Lecture Notes in Computer Science.

[28]  Ivan E. Sutherland,et al.  Sketchpad a man-machine graphical communication system , 1988, DAC 1988.

[29]  Ivan E. Sutherland,et al.  Sketchpad a Man-Machine Graphical Communication System , 1899, Outstanding Dissertations in the Computer Sciences.

[30]  Sumit Gulwani,et al.  A practical framework for constructing structured drawings , 2014, IUI.

[31]  Rastislav Bodík,et al.  Programming by manipulation for layout , 2014, UIST.

[32]  Daniel C. Halbert,et al.  Programming by Example , 2010, Encyclopedia of Machine Learning.

[33]  Anthony C. Davison,et al.  An introduction to the bootstrap with applications in R , 2002 .

[34]  Ivan E. Sutherland,et al.  Sketch pad a man-machine graphical communication system , 1964, DAC.

[35]  Ben Shneiderman,et al.  Direct Manipulation: A Step Beyond Programming Languages , 1983, Computer.