Programming by voice: a hands-free approach for motorically challenged children

This paper introduces a voice-driven tool applied to an Initial Programming Environment (IPE), which gives motorically challenged individuals the opportunity to learn programming skills; in particular, our project allows programming by voice within Scratch. Although the native Scratch environment allows users to create a program by arranging graphical blocks logically, such visual languages are completely dependent on the use of a mouse and keyboard. This modality of interaction limits users based on physical abilities. Our solution is a tool, called Myna, which is a voice-driven Java application executed parallel to Scratch. Myna processes voice commands from the user, interprets those commands according to a pre-defined grammar, and simulates synonymous actions of a mouse and keyboard within Scratch. The resulting environment assists those with a motor disability (particularly young children) in learning the joy of programming. This extended abstract describes the motivation behind the project, a technical description of Myna, and defines the current work in progress.

[1]  Alin Stefanescu,et al.  An empirical study of the state of the practice and acceptance of model-driven engineering in four industrial cases , 2013, Empirical Software Engineering.

[2]  Jesús Gallardo,et al.  CIAT-GUI: A MDE-compliant environment for developing Graphical User Interfaces of information systems , 2012, Adv. Eng. Softw..

[3]  Porfirio Tramontana,et al.  Using GUI ripping for automated testing of Android applications , 2012, 2012 Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering.

[4]  Jeffrey G. Gray,et al.  Teaching CS principles with app inventor , 2012, ACM-SE '12.

[5]  Krishnendu Roy,et al.  App inventor for android: report from a summer camp , 2012, SIGCSE '12.

[6]  Suleyman Uludag,et al.  Implementing IT0/CS0 with scratch, app inventor forandroid, and lego mindstorms , 2011, SIGITE '11.

[7]  Tom Yeh,et al.  Associating the visual representation of user interfaces with their internal structures and metadata , 2011, UIST.

[8]  Tsung-Hsiang Chang,et al.  Using graphical representation of user interfaces as visual references , 2011, UIST '11 Adjunct.

[9]  Qusay H. Mahmoud,et al.  Best practices in teaching mobile application development , 2011, ITiCSE '11.

[10]  Krzysztof Z. Gajos,et al.  Ability-Based Design: Concept, Principles and Examples , 2011, TACC.

[11]  Susan Loveland,et al.  Human computer interaction that reaches beyond desktop applications , 2011, SIGCSE.

[12]  Michael P. Rogers,et al.  Smart smartphone development: iOS versus android , 2011, SIGCSE.

[13]  James B. Fenwick,et al.  Teaching mobile computing and developing software to support computer science education , 2011, SIGCSE.

[14]  David Wolber,et al.  App inventor and real-world motivation , 2011, SIGCSE.

[15]  João Pascoal Faria,et al.  A metamodel-based approach for automatic user interface generation , 2010, MODELS'10.

[16]  Polona Vilar,et al.  Designing the User Interface: Strategies for Effective Human-Computer Interaction (5th edition) , 2010, J. Assoc. Inf. Sci. Technol..

[17]  Rob Miller,et al.  GUI testing using computer vision , 2010, CHI.

[18]  Morgan Dixon,et al.  Prefab: implementing advanced behaviors using pixel-based reverse engineering of interface structure , 2010, CHI.

[19]  Judith Gal-Ezer,et al.  Computer science teacher preparation is critical , 2010, INROADS.

[20]  Eric Rosenbaum,et al.  Scratch: programming for all , 2009, Commun. ACM.

[21]  Rob Miller,et al.  Sikuli: using GUI screenshots for search and automation , 2009, UIST '09.

[22]  Vinay Kulkarni,et al.  Model Driven Development of Graphical User Interfaces for Enterprise Business Applications - Experience, Lessons Learnt and a Way Forward , 2009, MoDELS.

[23]  Jeff A. Bilmes,et al.  Longitudinal study of people learning to use continuous voice-based cursor control , 2009, CHI.

[24]  Kimiko Ryokai,et al.  Multimodal programming environment for kids: a "thought bubble" interface for the Pleo robotic character , 2009, CHI Extended Abstracts.

[25]  Susan H. Rodger,et al.  Engaging middle school teachers and students with alice in a diverse set of subjects , 2009, SIGCSE '09.

[26]  Atif M. Memon,et al.  Automatically repairing event sequence-based GUI test suites for regression testing , 2008, TSEM.

[27]  Gregg C. Vanderheiden,et al.  Ubiquitous Accessibility, Common Technology Core, and Micro Assistive Technology: Commentary on “Computers and People with Disabilities” , 2008, TACC.

[28]  Richard E. Ladner,et al.  Access and Empowerment: Commentary on “Computers and People with Disabilities” , 2008, TACC.

[29]  Alistair D. N. Edwards,et al.  Keeping Up with Technology: Commentary on “Computers and People with Disabilities” , 2008, TACC.

[30]  Alan F. Newell,et al.  Accessible Computing -- Past Trends and Future Suggestions: Commentary on “Computers and People with Disabilities” , 2008, TACC.

[31]  Renata Pontin de Mattos Fortes,et al.  Multimodal interfaces design issues: the fusion of well-designed voice and graphical user interfaces , 2008, SIGDOC '08.

[32]  Philip Kortum,et al.  HCI Beyond the GUI: Design for Haptic, Speech, Olfactory, and Other Nontraditional Interfaces , 2008 .

[33]  Débora Nice Ferrari Barbosa,et al.  Local: a model geared towards ubiquitous learning , 2008, SIGCSE '08.

[34]  Aniruddha S. Gokhale,et al.  Domain-Specific Modeling , 2008, Handbook of Dynamic System Modeling.

[35]  Stefan Staiger,et al.  Reverse Engineering of Graphical User Interfaces Using Static Analyses , 2007, 14th Working Conference on Reverse Engineering (WCRE 2007).

[36]  R. Smith,et al.  An Overview of the Tesseract OCR Engine , 2007, Ninth International Conference on Document Analysis and Recognition (ICDAR 2007).

[37]  Mitchel Resnick,et al.  All I really need to know (about creative thinking) I learned (by studying how children learn) in kindergarten , 2007, C&C '07.

[38]  Becky Gibson,et al.  Enabling an accessible web 2.0 , 2007, W4A '07.

[39]  David J. Malan,et al.  Scratch for budding computer scientists , 2007, SIGCSE.

[40]  Joseph S. Valacich,et al.  Fine-Tuning the Human-Computer Interface: Verbal versus Keyboard Input in an Idea Generation Context , 2007, 2007 40th Annual Hawaii International Conference on System Sciences (HICSS'07).

[41]  Shari Trewin,et al.  Physical usability and the mobile web , 2006, W4A '06.

[42]  Elsbeth M. de Korte,et al.  The effect of speech recognition on working postures, productivity and the perception of user friendliness. , 2006 .

[43]  Alain Désilets,et al.  VoiceCode: an innovative speech interface for programming-by-voice , 2006, CHI Extended Abstracts.

[44]  Daniela Marghitu,et al.  Educational and Technological Computer Literacy Program for Typical and Special Needs Children: An Auburn University Case Study , 2006 .

[45]  Jeffrey G. Gray,et al.  Automating change evolution in model-driven engineering , 2006, Computer.

[46]  Caitlin Kelleher,et al.  Lowering the barriers to programming: A taxonomy of programming environments and languages for novice programmers , 2005, CSUR.

[47]  Barbara Ericson,et al.  Introduction to Computing and Programming with Java: A Multimedia Approach , 2005 .

[48]  Randy Allen Harris,et al.  Voice Interaction Design: Crafting the New Conversational Speech Systems , 2004 .

[49]  Andrew Sears,et al.  Speech-based cursor control: a study of grid-based solutions , 2003, ASSETS.

[50]  D. Wilson.  Multimodal programming for dyslexic students , 2004, International Conference on Multimodal Interaction.

[51]  Jennifer Balogh,et al.  Voice User Interface Design , 2004 .

[52]  Mitchel Resnick,et al.  Scratch: A Sneak Preview , 2004 .

[53]  Faizan Javed,et al.  SpeechClipse: an Eclipse speech plug-in , 2003, eclipse '03.

[54]  Clare-Marie Karat,et al.  Hands-Free, Speech-Based Navigation During Dictation: Difficulties, Consequences, and Solutions , 2003, Hum. Comput. Interact..

[55]  Avraham Leff,et al.  Web-application development using the Model/View/Controller design pattern , 2001, Proceedings Fifth IEEE International Enterprise Distributed Object Computing Conference.

[56]  Alan F. Blackwell,et al.  Visual Programming in the Wild: A Survey of LabVIEW Programmers , 2001, J. Vis. Lang. Comput..

[57]  Laila Dybkjær,et al.  Usability Evaluation in Spoken Language Dialogue Systems , 2001, ACL 2001.

[58]  M J Haxer,et al.  Use of speech recognition software: a vocal endurance test for the new millennium? , 2001, Journal of voice : official journal of the Voice Foundation.

[59]  Sharon L. Oviatt,et al.  Taming recognition errors with a multimodal interface , 2000, CACM.

[60]  Luke S. Zettlemoyer,et al.  A visual medium for programmatic control of interactive applications , 1999, CHI '99.

[61]  Sharon L. Oviatt,et al.  The efficiency of multimodal interaction: a case study , 1998, ICSLP.

[62]  Ben Shneiderman,et al.  Designing the User Interface: Strategies for Effective Human-Computer Interaction , 1998 .

[63]  Kirsten N. Whitley.  Visual Programming Languages and the Empirical Evidence For and Against , 1997, J. Vis. Lang. Comput..

[64]  Marian Petre,et al.  Usability Analysis of Visual Programming Environments: A 'Cognitive Dimensions' Framework , 1996, J. Vis. Lang. Comput..

[65]  Alistair D. N. Edwards,et al.  An approach to the evaluation of assistive technology , 1996, Assets '96.

[66]  T. V. Raman,et al.  Emacspeak—direct speech access , 1996, Assets '96.

[67]  Sheryl Burgstahler,et al.  Increasing the Representation of People with Disabilities in Science, Engineering and Mathematics , 1994 .

[68]  Alan C. Kay.  The early history of Smalltalk , 1993, HOPL-II.

[69]  B. W. York,et al.  Computers and people with disabilities , 1992, CACM.

[70]  Alexander I. Rudnicky,et al.  A Comparison of Speech and Typed Input , 1990, HLT.

[71]  Gale Martin,et al.  The Utility of Speech Input in User-Computer Interfaces , 1989, Int. J. Man Mach. Stud..

[72]  Seymour Papert,et al.  Mindstorms: Children, Computers, and Powerful Ideas , 1981 .

[73]  Richard A. Bolt,et al.  “Put-that-there”: Voice and gesture at the graphics interface , 1980, SIGGRAPH '80.

[74]  Harold Abelson,et al.  App Inventor - Create Your Own Android Apps , 2011 .

[75]  Jason Alexander,et al.  Computer science unplugged: school students doing real computing without computers , 2009 .

[76]  Roy D. Pea,et al.  Logo Programming and Problem Solving , 2007 .

[77]  Atif M. Memon,et al.  Automating regression testing for evolving GUI software , 2005, J. Softw. Maintenance Res. Pract..

[78]  Glen Bull,et al.  Children, Computers, and Powerful Ideas. , 2005 .

[79]  F. Hobbs,et al.  Age and Sex Composition , 2004 .

[80]  Cme Materials,et al.  The National Institute of Neurological Disorders and Stroke of the National Institutes of Health , 2002 .

[81]  J. Nietupski,et al.  Preparing students with mild disabilities for careers in technology: A process and recommendations from Iowa's High School High Tech program , 2001 .

[82]  Damian Conway,et al.  GRAIL: A Zeroth Programming Language , 1999 .

[83]  Shari Trewin,et al.  Dynamic Modelling of Keyboard Skills: Supporting Users With Motor Disabilities , 1997 .

[84]  Ralph Johnson,et al.  design patterns elements of reusable object oriented software , 2019 .