Interactive sketching for the early stages of user interface design

Current interactive user interface construction tools are often more of a hindrance than a benefit during the early stages of user interface design. These tools take too much time to use and force designers to specify more of the design details than they wish to at this early stage. Most interface designers, especially those who have a background in graphic design, prefer to sketch early interface ideas on paper or on a white-board. We are developing an interactive tool that allows designers to quickly sketch an interface using an electronic pad and stylus. Our tool preserves the important properties of paper: A rough drawing can be produced very quickly and the medium is very flexible. However, unlike a paper sketch this electronic sketch can easily be exercised and modified. In addition, our system allows designers to examine, annotate, and edit a complete history of the design. When the designer is satisfied with this early prototype, the system can transform the sketch into a complete, finished interface in a specified look-and-feel. This transformation takes place with the guidance of the designer. By supporting the early design phases of the software life-cycle, our tool should both ease the prototyping of user interfaces and improve the speed with which a final interface can be created.

[1]  Shi-Kuo Chang,et al.  Visual Languages , 1986, Management and Information Systems.

[2]  Catherine G. Wolf A Comparative Study of Gestural and Keyboard Interfaces , 1988 .

[3]  S. Joy Mountford,et al.  The Art of Human-Computer Interface Design , 1990 .

[4]  Rui Zhao Handsketch-based diagram editing , 1993 .

[5]  Takayuki Dan Kimura,et al.  A comparison study of the pen and the mouse in editing graphic diagrams , 1993, Proceedings 1993 IEEE Symposium on Visual Languages.

[6]  J. Fodor,et al.  The Language of Thought , 1980 .

[7]  Eric J. Golin,et al.  Early Experience with the Visual Programmer's WorkBench , 1990, IEEE Trans. Software Eng..

[8]  Marc Rettig,et al.  Prototyping for tiny fingers , 1994, CACM.

[9]  Ian H. Witten,et al.  Metamouse: specifying graphical procedures by example , 1989, SIGGRAPH.

[10]  Yin Yin Wong Rough and ready prototypes: lessons from graphic design , 1992, CHI '92.

[11]  Charles C. Tappert,et al.  Online recognizer for runon handprinted characters , 1990, [1990] Proceedings. 10th International Conference on Pattern Recognition.

[12]  Fumihiko Kimura,et al.  An Interactive Geometrical Design System with Handwriting Input , 1977, IFIP Congress.

[13]  Eugene Charniak,et al.  Bayesian Networks without Tears , 1991, AI Mag..

[14]  Roger B. Dannenberg,et al.  Creating graphical interactive application objects by demonstration , 1989, UIST '89.

[15]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[16]  Herbert A. Simon,et al.  Why a Diagram is (Sometimes) Worth Ten Thousand Words , 1987 .

[17]  Jonathan Stephen Fish,et al.  Amplifying the Mind’s Eye: Sketching and Visual Cognition , 1990 .

[18]  Thomas P. Moran,et al.  Tivoli: an electronic whiteboard for informal workgroup meetings , 1993, INTERCHI.

[19]  Clive Frankish,et al.  Recognition accuracy and user acceptance of pen interfaces , 1995, CHI '95.

[20]  John K. Ousterhout,et al.  An X11 Toolkit Based on the Tcl Language , 1991, USENIX Winter.

[21]  Yin Yin Wong Layer tool: support for progressive design , 1993, CHI '93.

[22]  Mary LaLomia User acceptance of handwritten recognition accuracy , 1994, CHI '94.

[23]  Luca Cardelli,et al.  Building user interfaces by direct manipulation , 1988, UIST '88.

[24]  James R. Rhyne,et al.  The paper-like interface , 1989 .

[25]  Francesmary Modugno Extending end-user programming in a visual shell with programming by demonstration and graphical lan , 1995 .

[26]  Brad A. Myers,et al.  Taxonomies of visual programming and program visualization , 1990, J. Vis. Lang. Comput..

[27]  James A. Landay,et al.  Sketching storyboards to illustrate interface behaviors , 1996, CHI Conference Companion.

[28]  F. Thomas,et al.  The illusion of life : Disney animation , 1981 .

[29]  Dan Shafer,et al.  The power of PenPoint , 1991 .

[30]  Thomas P. Moran,et al.  Implicit structure for pen-based systems within a freeform interaction paradigm , 1995, CHI '95.

[31]  Brad A. Myers,et al.  Demonstrational interfaces: A step beyond direct manipulation , 1992, Computer.

[32]  David Wolber,et al.  Adding rule-based reasoning to a demonstrational interface builder , 1992, UIST '92.

[33]  Brad A. Myers,et al.  Creating user interfaces by demonstration , 1988 .

[34]  James A. Landay,et al.  Inferring graphical constraints with Rockit , 1993, INTERCHI.

[35]  David Canfield Smith,et al.  KidSim: end user programming of simulations , 1995, CHI '95.

[36]  James D. Herbsleb,et al.  Preserving knowledge in design projects: what designers need to know , 1993, INTERCHI.

[37]  Jonathan Meyer EtchaPad—disposable sketch based interfaces , 1996, CHI Conference Companion.

[38]  Gurminder Singh,et al.  Druid: a system for demonstrational rapid user interface development , 1990, UIST '90.

[39]  Graham I. Johnson,et al.  Evaluating usability of human computer interfaces: a practical method (west sussex , 1989 .

[40]  Ilse M. Verstijnen,et al.  Considerations for electronic idea-creation tools , 1996, CHI Conference Companion.

[41]  Robert J. K. Jacob,et al.  A Specification Language for Direct-Manipulation User Interfaces , 1986, ACM Trans. Graph..

[42]  Brad A. Myers,et al.  Marquise: creating complete user interfaces by demonstration , 1993, CHI '93.

[43]  Thomas Strothotte,et al.  Assessing the effect of non-photorealistic rendered images in CAD , 1996, CHI '96.

[44]  Dan Venolia,et al.  T-Cube: a fast, self-disclosing pen-based alphabet , 1994, CHI '94.

[45]  M. Berthod,et al.  Automatic recognition of handprinted characters—The state of the art , 1980, Proceedings of the IEEE.

[46]  Henry Lieberman,et al.  Watch what I do: programming by demonstration , 1993 .

[47]  Eric J. Golin A method for the specification and parsing of visual languages , 1991 .

[48]  David Salesin,et al.  Computer-generated pen-and-ink illustration , 1994, SIGGRAPH.

[49]  F. Thomas,et al.  Disney Animation: The Illusion of Life , 1981 .

[50]  Takayuki Dan Kimura,et al.  Recognizing multistroke geometric shapes: an experimental evaluation , 1993, UIST '93.

[51]  G. W. Furnas,et al.  Generalized fisheye views , 1986, CHI '86.

[52]  Brad A. Myers,et al.  User interface software tools , 1995, TCHI.

[53]  D. A. Henderson,et al.  The Trillium user interface design environment , 1986, CHI '86.

[54]  David Goldberg,et al.  Touch-typing with a stylus , 1993, INTERCHI.

[55]  Pierre D. Wellner,et al.  Statemaster: A UIMS based on statecharts for prototyping and target implementation , 1989, CHI '89.

[56]  Raghu Kolli,et al.  Deriving the Functional Requirements for a Concept Sketching Device: A Case Study , 1993, VCHCI.

[57]  Brad A. Myers,et al.  Graphical representation and feedback in a PBD system , 1993 .

[58]  Daniel Boyarski,et al.  Computers and communication design: exploring the rhetoric of HCI , 1994, INTR.

[59]  T. O. Ellis,et al.  The RAND tablet: a man-machine graphical communication device , 1964, AFIPS '64 (Fall, part I).

[60]  Eric Saund,et al.  A perceptually-supported sketch editor , 1994, UIST '94.

[61]  Axel Kramer,et al.  Translucent patches—dissolving windows , 1994, UIST '94.

[62]  Rui Zhao,et al.  Incremental recognition in gesture-based and syntax-directed diagram editors , 1993, INTERCHI.

[63]  Nan C. Shu,et al.  Visual Programming: Perspectives and Approaches , 1989, IBM Syst. J..

[64]  Axel Kramer,et al.  Translucent history , 1995, CHI '95.

[65]  V. Goel Sketches of thought , 1995 .

[66]  Dean Rubine,et al.  The automatic recognition of gestures , 1992 .

[67]  Nicholas Negroponte,et al.  Recent advances in sketch recognition , 1973, AFIPS National Computer Conference.

[68]  Wayne Citrin,et al.  Diagram entry mechanisms in graphical environments , 1995, CHI '95.

[69]  Donald A. Norman,et al.  User Centered System Design: New Perspectives on Human-Computer Interaction , 1988 .

[70]  J Rhyne Dialogue management for gestural interfaces , 1987, COMG.

[71]  Herbert A. Simon,et al.  Why a diagram is (sometimes) worth 10, 000 word , 1987 .

[72]  Ping Luo,et al.  Facilitating the exploration of interface design alternatives: the HUMANOID model of interface design , 1992, CHI.

[73]  Ellen Yi-Luen Do,et al.  Ambiguous intentions: a paper-like interface for creative design , 1996, UIST '96.

[74]  Ching Y. Suen,et al.  The State of the Art in Online Handwriting Recognition , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[75]  Paula M. Ferguson Motif reference manual , 1993 .

[76]  William Buxton,et al.  Creating highly-interactive and graphical user interfaces by demonstration , 1986, SIGGRAPH.

[77]  Mark Weiser,et al.  Some computer science issues in ubiquitous computing , 1993, CACM.

[78]  John C. Tang,et al.  Liveboard: a large interactive display supporting group meetings, presentations, and remote collaboration , 1992, CHI.

[79]  David Harel,et al.  Statecharts: A Visual Formalism for Complex Systems , 1987, Sci. Comput. Program..

[80]  John Lansdown,et al.  Expressive rendering: a review of nonphotorealistic techniques , 1995, IEEE Computer Graphics and Applications.

[81]  Wayne Citrin,et al.  Requirements for graphical front ends for visual languages , 1993, Proceedings 1993 IEEE Symposium on Visual Languages.

[82]  Mark Weiser,et al.  Some Computer Science Problems in Ubiquitous Computing , 1993 .

[83]  Roger B. Dannenberg,et al.  Garnet: comprehensive support for graphical, highly interactive user interfaces , 1995 .

[84]  Mark D. Gross,et al.  Recognizing and interpreting diagrams in design , 1994, AVI '94.

[85]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[86]  David Pugh Using Interactive Sketch Interpretation to Design Solid Objects , 1993 .

[87]  Theodosios Pavlidis,et al.  An automatic beautifier for drawings and illustrations , 1985, SIGGRAPH.

[88]  Ivan E. Sutherland,et al.  Sketchpad a Man-Machine Graphical Communication System , 1899, Outstanding Dissertations in the Computer Sciences.

[89]  Shi-Kuo Chang,et al.  Visual Languages: A Tutorial and Survey , 1986, IEEE Software.

[90]  Rui Zhao On-Line Geometry Recognition Using C++, an Object-Oriented Approach , 1992, TOOLS.

[91]  Henry Lieberman,et al.  Mondrian: a teachable graphical editor , 1993, INTERCHI.

[92]  David Salesin,et al.  Interactive pen-and-ink illustration , 1994, SIGGRAPH.

[93]  Mark D. Gross,et al.  Distributed architectures for pen-based input and diagram recognition , 1996, AVI '96.

[94]  D. B. Davis,et al.  Sun Microsystems Inc. , 1993 .

[95]  Alison Black Visible planning on paper and on screen: The impact of working medium on decision-making by novice graphic designers , 1990 .

[96]  William Buxton,et al.  Issues in combining marking and direct manipulation techniques , 1991, UIST '91.

[97]  M. Weiser,et al.  An empirical comparison of pie vs. linear menus , 1988, CHI '88.

[98]  David R. Forsey,et al.  How to Render Frames and Influence People , 1994, Comput. Graph. Forum.

[99]  James A. Landay,et al.  Just Draw It! Programming by Sketching Storyboards, , 1995 .

[100]  Dean Rubine,et al.  Specifying gestures by example , 1991, SIGGRAPH.

[101]  Bill Buxton,et al.  GEdit: a test bed for editing by contiguous gestures , 1991, SGCH.

[102]  W. Wang,et al.  A Survey of 3D Solid Reconstruction from 2D Projection Line Drawings , 1993, Comput. Graph. Forum.

[103]  Brad A. Myers Separating application code from toolkits: eliminating the spaghetti of call-backs , 1991, UIST '91.

[104]  Fernando Nunes Ferreira,et al.  Sketching 3D models with 2D interaction devices , 1994, Comput. Graph. Forum.

[105]  Steven K. Feiner,et al.  Graphical editing by example , 1993 .

[106]  David Wolber,et al.  A demonstrational technique for developing interfaces with dynamically created objects , 1991, UIST '91.

[107]  William Buxton,et al.  Towards a comprehensive user interface management system , 1983, SIGGRAPH.

[108]  William Buxton,et al.  There's more to interaction than meets the eye: some issues in manual input , 1987 .

[109]  Fred Lakin,et al.  Spatial Parsing for Visual Languages , 1986 .

[110]  Clayton Lewis,et al.  Designing for usability—key principles and what designers think , 1983, CHI '83.

[111]  Godfried T. Toussaint,et al.  Results Obtained Using a Simple Character Recognition Procedure on Munson's Handprinted Data , 1972, IEEE Transactions on Computers.

[112]  Brad A. Myers A new model for handling input , 1990, TOIS.

[113]  James D. Hollan,et al.  Pad++: a zooming graphical interface for exploring alternate interface physics , 1994, UIST '94.

[114]  Roger B. Dannenberg,et al.  Garnet: comprehensive support for graphical, highly interactive user interfaces , 1990, Computer.

[115]  Alexander Repenning,et al.  Agentsheets: a tool for building domain-oriented dynamic, visual environments , 1993 .

[116]  Alexander Repenning,et al.  Agentsheets: a tool for building visual programming environments , 1992, CHI '92.

[117]  James A. Landay,et al.  SILK: sketching interfaces like krazy , 1996, CHI Conference Companion.

[118]  Martin Odersky,et al.  Building visual language parsers , 1991, CHI '91.

[119]  E.R. Brocklehurst The NPL Electronic Paper Project , 1991, Int. J. Man Mach. Stud..

[120]  Daniel C. Halbert,et al.  Programming by Example , 2010, Encyclopedia of Machine Learning.

[121]  James A. Landay Tools review: Serius - a visual programming environment , 1991, J. Vis. Lang. Comput..

[122]  James A. Landay,et al.  Extending an existing user interface toolkit to support gesture recognition , 1993, INTERCHI Adjunct Proceedings.