Applying computational semantics to the real-time communication of skill knowledge

Global skill shortages are reported in many occupations. Existing strategies for addressing skill shortages are not successful and, as a result, skill shortages are an intractable problem. A new strategy, real-time communication of skill knowledge without human instructors, has the potential to bring about radical reductions in skill shortages. The goal of the study reported in this VTT Working Paper was to determine how skill knowledge could be communicated in real-time without reliance on access to a person with relevant existing skill knowledge. In particular, the communication of manual skills. The research involved literature review and field study. Literature review encompassed studies concerned with skill knowledge, communication media, and computational semantics. Field study involved interviews with industry practitioners seeking to address skill shortages and computational semantics scientists. The study revealed that many of the technologies and methods required for the realtime communication of skill knowledge without human instructors are already available. Further, the study revealed that computational semantics is essential to the successful application and integration of these technologies and methods. ISBN 978-951-38-7162-8 (URL: http://www.vtt.fi/publications/index.jsp) Series title and ISSN Project number VTT Working Papers 1459-7683 (URL: http://www.vtt.fi/publications/index.jsp) 21791 Date Language Pages June 2008 English 85 p. Name of project Commissioned by Rapid Economic Production of Special Products VTT, Tekes, companies

[1]  Lars Leden,et al.  Safe and joyful cycling for senior citizens , 2007 .

[2]  E. A. Fleishman,et al.  Role of kinesthetic and spatialvisual abilities in perceptual-motor learning. , 1963, Journal of experimental psychology.

[3]  James A. Hendler,et al.  Spinning the Semantic Web: Bringing the World Wide Web to Its Full Potential , 2002 .

[4]  Tuomas Ihme,et al.  Challenges of software-hardware co-design: Prestudy in TWINS project , 2008 .

[5]  Carolyn Copper,et al.  Does mental practice enhance performance , 1994 .

[6]  Carlos Pedrinaci,et al.  Towards a semantic event-based service-oriented architecture , 2006 .

[7]  Totti Könnölä,et al.  System Transition - Concepts and Framework for Analysing Nordic Energy System Research and Governance , 2008 .

[8]  Thierry Pun,et al.  A contextual model for semantic video structuring , 2005, 2005 13th European Signal Processing Conference.

[9]  R. Snow,et al.  Implications of cognitive psychology for educational measurement. , 1989 .

[10]  Malcolm S. Cohen,et al.  Global Skill Shortages , 2002 .

[11]  Andrea Gaggioli,et al.  Virtual Reality Training for Health-Care Professionals , 2003, Cyberpsychology Behav. Soc. Netw..

[12]  P. Hagoort.  On Broca, brain, and binding: a new framework , 2005, Trends in Cognitive Sciences.

[13]  Martin Fischer,et al.  Using 3D and 4D models to improve jobsite communication – virtual huddles case study , 2007 .

[14]  Charles M. Reigeluth,et al.  Instructional-Design Theories and Models: A New Paradigm of Instructional Theory , 1999 .

[15]  W. Shebilske,et al.  Motor Learning and Control , 1993 .

[16]  Y. Fery.  Differentiating visual and kinesthetic imagery in mental practice. , 2003, Canadian journal of experimental psychology = Revue canadienne de psychologie experimentale.

[17]  Vladan Devedzic,et al.  Understanding ontological engineering , 2002, CACM.

[18]  Riitta Kyrki-Rajamäki,et al.  Finnish research network for generation four nuclear energy systems , 2008 .

[19]  Victor Zue,et al.  Conversational interfaces: advances and challenges , 1997, Proceedings of the IEEE.

[20]  Michèle Courant,et al.  A Generic Coordination Model for Pervasive Computing Based on Semantic Web Languages , 2004, NLDB.

[21]  David M. Lilien,et al.  Sectoral Shifts and Cyclical Unemployment , 1982, Journal of Political Economy.

[22]  Marja Liinasuo,et al.  Kuution kokoaminen AR-teknologian avulla , 2008 .

[23]  Gerd Stumme,et al.  Conceptual Structures: Common Semantics for Sharing Knowledge. Proc. , 2005 .

[24]  Harry Bunt,et al.  Standardization in Multimodal Content Representation: Some Methodological Issues , 2004, LREC.

[25]  Gabriele Wulf,et al.  Self-Controlled Feedback Is Effective if It Is Based on the Learner's Performance , 2005, Research quarterly for exercise and sport.

[26]  R N Singer,et al.  Subject-controlled performance feedback and learning of a closed motor skill. , 1995, Perceptual and motor skills.

[27]  Marc R. Dickey,et al.  A Comparison of Verbal Instruction and Nonverbal Teacher-Student Modeling in Instrumental Ensembles , 1991 .

[28]  M. Latash,et al.  Motor control and learning , 2006 .

[29]  Sharon Flank.  The role of natural-language processing in multimedia , 1995, CSUR.

[30]  Song Mao,et al.  Stochastic language model for analyzing document physical layout , 2001, IS&T/SPIE Electronic Imaging.

[31]  L K Tennant,et al.  Maximizing performance feedback effectiveness through videotape replay and a self-controlled learning environment. , 1997, Research quarterly for exercise and sport.

[32]  Gabriele Wulf,et al.  Feedback After Good Trials Enhances Learning , 2007, Research quarterly for exercise and sport.

[33]  James R. Curran,et al.  Improving the Efficiency of a Wide-Coverage CCG Parser , 2007, IWPT.

[34]  Miikka Tikander,et al.  SOUND QUALITY OF AN AUGMENTED REALITY AUDIO HEADSET , 2005 .

[35]  Carlous Caple,et al.  The effects of spaced practice and spaced review on recall and retention using computer-assisted instruction , 1996 .

[36]  Charles H. Shea,et al.  Understanding the role of augmented feedback : The good, the bad and the ugly , 2004 .

[37]  Harry Bunt,et al.  Dimensions in Dialogue Act Annotation , 2006, LREC.

[38]  Barbara Tversky,et al.  Communicative Gestures Facilitate Problem Solving for Both Communicators and Recipients , 2007, Model-Based Reasoning in Science, Technology, and Medicine.

[39]  J. R. Wittenborn.  Appraising vocational fitness. , 1950 .

[40]  H. H. Kaufman,et al.  Teaching surgeons to operate—Principles of psychomotor skills training , 2005, Acta Neurochirurgica.

[41]  L. S. Vygotskiĭ,et al.  Mind in society : the development of higher psychological processes , 1978 .

[42]  N. Callow,et al.  The relationship between the use of kinaesthetic imagery and different visual imagery perspectives , 2004, Journal of sports sciences.

[43]  R. Sternberg.  The Triarchic Mind: A New Theory of Human Intelligence , 1988 .

[44]  Lars Bretzner,et al.  A Prototype System for Computer Vision Based Human Computer Interaction , 2001 .

[45]  Barbara J. McNeil,et al.  Meta-analysis of interactive video instruction: A 10 year review of achievement effects. , 1991 .

[46]  Richard Stottler,et al.  Designing and Implementing Intelligent Tutoring Instruction for Tactical Action Officers , 2007 .

[47]  Erkki Lehtinen,et al.  Vesihuoltolaitosten verkosto-omaisuuden hallinta: Toimintamallin kuvaus Total Management Planning -ohjeistuksen pohjalta , 2008 .

[48]  Thomas B. Moeslund,et al.  A Survey of Computer Vision-Based Human Motion Capture , 2001, Comput. Vis. Image Underst..

[49]  Ilkka Niemelä,et al.  NPP Safety Automation Systems Analysis: State of the Art , 2008 .

[50]  Juan Carlos Augusto,et al.  Ambient Intelligence: The Confluence of Ubiquitous/Pervasive Computing and Artificial Intelligence , 2007 .

[51]  Stanley Peters,et al.  CHAT to Your Destination , 2007, SIGDIAL.

[52]  David L. Tennenhouse,et al.  Proactive computing , 2000, Commun. ACM.

[53]  Asunción Gómez-Pérez,et al.  Building a chemical ontology using Methontology and the Ontology Design Environment , 1999, IEEE Intell. Syst..

[54]  Xunlei Rose Hu rosehu.  A survey of machine learning approaches to analysis of large corpora , 2003 .

[55]  Meredyth Krych Appelbaum,et al.  When Coordination is Worth a Thousand Words: the Role of Gesture in Grounding , 2004 .

[56]  R. Schmidt,et al.  Knowledge of results and motor learning: a review and critical reappraisal. , 1984, Psychological bulletin.

[57]  Douglas E. Appelt,et al.  GEMINI: A Natural Language System for Spoken-Language Understanding , 1993, ACL.

[58]  Lawrence F. Katz,et al.  Cyclical Unemployment: Sectoral Shifts or Aggregate Disturbances? , 1984, Journal of Political Economy.

[59]  Michael Milligan,et al.  Design and operation of power systems with large amounts of wind power: State of the art report , 2007 .

[60]  E. Simpson.  THE CLASSIFICATION OF EDUCATIONAL OBJECTIVES, PSYCHOMOTOR DOMAIN. , 1966 .

[61]  Jennifer L. Schwartz,et al.  An education theory-based method to teach a procedural skill. , 2004, Archives of dermatology.

[62]  Fiorella Padoa Schioppa,et al.  Mismatch in Japan , 1991 .

[63]  Stacy Marsella,et al.  Building Interactive Virtual Humans for Training Environments , 2007 .

[64]  Albrecht Schmidt,et al.  Towards distributed awareness - an artifact based approach , 2004, Sixth IEEE Workshop on Mobile Computing Systems and Applications.

[65]  Timothy L.J. Ferris,et al.  A PSYCHOMOTOR SKILLS EXTENSION TO BLOOM'S TAXONOMY OF EDUCATION OBJECTIVES FOR ENGINEERING EDUCATION , 2005 .

[66]  Deborah L. McGuinness,et al.  Ontologies Come of Age , 2003, Spinning the Semantic Web.

[67]  Martin Erwig.  Visual semantics-or: what you see is what you compute , 1998, Proceedings. 1998 IEEE Symposium on Visual Languages (Cat. No.98TB100254).

[68]  R. Cucchiara,et al.  Object-based and event-based semantic video adaptation , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[69]  Andrew Salvatore Gentile.  Effects of symbolic modeling and behaviour rehearsal on assertive training with prison inmates , 1976 .

[70]  E Bizzi,et al.  Augmented Feedback Presented in a Virtual Environment Accelerates Learning of a Difficult Motor Task. , 1997, Journal of motor behavior.

[71]  John Haynes,et al.  Assessing the rigor and relevance of planned instruction on an authentic assessment task in which pre-service teachers target the moving skills of the NSW PDHPEK-6 syllabus. , 2006 .

[72]  Brady Clark,et al.  Responding to Student Uncertainty in Spoken Tutorial Dialogue Systems , 2006, Int. J. Artif. Intell. Educ..

[73]  Ching-chih Chen,et al.  Using sharable ontology to retrieve historical images , 2002, JCDL '02.

[74]  Francis Jeffry Pelletier,et al.  Representation and Inference for Natural Language: A First Course in Computational Semantics , 2005, Computational Linguistics.

[75]  Jong Kyoung Kim,et al.  Speech recognition , 1983, 1983 IEEE International Solid-State Circuits Conference. Digest of Technical Papers.

[76]  Shih-Fu Chang,et al.  Discovering recurrent visual semantics in consumer photographs , 2000, Proceedings 2000 International Conference on Image Processing (Cat. No.00CH37101).

[77]  S. L. Tanimoto,et al.  Representation and learnability in visual languages for Web-based interpersonal communication , 1997, Proceedings. 1997 IEEE Symposium on Visual Languages (Cat. No.97TB100180).

[78]  Robert W. M. de Maclay.  Recent developments in psychomotor domain taxonomies of educational objectives and their application to experimental science , 1974 .

[79]  Vladimir Estivill-castro,et al.  Patterns in Geographical Data via Maximum Entropy Classification , 1997 .

[80]  David H. Jonassen,et al.  Handbook of Task Analysis Procedures , 1989 .

[81]  C. Myers.  Educational Psychology , 1904, Nature.

[82]  Holger Kenn,et al.  Wearable Computing for Aircraft Maintenance: Simplifying the User Interface , 2006 .

[83]  Sue Dockett,et al.  Australian association for research in education , 1989 .

[84]  Jeff Rickel,et al.  Virtual Humans for Team Training in Virtual Reality , 1999 .

[85]  Max Mühlhäuser,et al.  Natural Language Processing for Ambient Intelligence , 2007, Künstliche Intell..

[86]  Steven J. Davis,et al.  Gross Job Creation, Gross Job Destruction and Employment Reallocation , 1991 .

[87]  Caroline Dean Hollingsworth.  comparison of verbal and nonverbal instruction in elementary school mathematics , 1973 .

[88]  G. A. Ferguson,et al.  HUMAN ABILITIES. , 1965, Annual review of psychology.

[89]  Thomas R. Gruber,et al.  A translation approach to portable ontology specifications , 1993 .

[90]  Olavi Lehtoranta,et al.  Effects of Public R&D Funding , 2008 .

[91]  Harold Pashler,et al.  The Effect of Overlearning on Long-Term Retention , 2005 .

[92]  R A Schmidt,et al.  Muscular fatigue and learning a discrete motor skill. , 1971, Research quarterly.

[93]  Antti Lehtilä,et al.  EU:n yksipuolinen päästöjen rajoittaminen ja kehittyneiden maiden yhteinen päästöjen rajoittaminen: Vaikutukset Suomeen arvioituna TIMES-mallilla , 2008 .

[94]  J Mount.  Effectiveness of Visual vs Kinesthetic Instruction for Learning a Gross Motor Skill , 1987, Perceptual and motor skills.

[95]  Sher ry Folsom-Meek,et al.  Human Performance , 1953, Nature.

[96]  Jan Rutkowski,et al.  Labor Markets in EU8+2: From the Shortage of Jobs to the Shortage of Skilled Workers , 2007 .

[97]  Florian Michahelles,et al.  Proactive Instructions for Furniture Assembly , 2002, UbiComp.

[98]  J. Bailenson,et al.  The Use of Immersive Virtual Reality in the Learning Sciences: Digital Transformations of Teachers, Students, and Social Context , 2008 .

[99]  Hui Guo,et al.  Stock market dispersion and unemployment , 2007 .

[100]  Ilkka Niemelä,et al.  Model-Based Analysis of an Arc Protection and an Emergency Cooling System , 2008 .

[101]  Richard A. Schmidt,et al.  Frequent Augmented Feedback Can Degrade Learning: Evidence and Interpretations , 1991 .

[102]  M. Mahoney,et al.  Psychology of the elite athlete: An exploratory study , 1977, Cognitive Therapy and Research.

[103]  Y A Féry,et al.  Kinesthetic and Visual Image in Modeling Closed Motor Skills: The Example of the Tennis Serve , 2000, Perceptual and motor skills.

[104]  Pertti Lahdenperä.  Innovaatioita edistämässä: Lähtökohtia ja ajatuksia rakennus- ja infra-alan hankintamallien kehittämiseen , 2007 .

[105]  Mohammed Belkhatir,et al.  A signal/semantic framework for image retrieval , 2005, Proceedings of the 5th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL '05).

[106]  Gibson F. Darden.  Videotape Feedback for Student Learning and Performance: A Learning-Stages Approach , 1999 .

[107]  Ruzena Bajcsy,et al.  The Effects of Fully Immersive Virtual Reality on the Learning of Physical Tasks , 2006 .

[108]  K Moorthy,et al.  Simulation based training , 2005, BMJ : British Medical Journal.

[109]  Ilkka Niemelä,et al.  Model-Based Analysis of an Arc Protection and an Emergency Cooling System - MODSAFE 2007 Working Report , 2008 .

[110]  P. Kirschner,et al.  Taking the Load Off a Learner's Mind: Instructional Design for Complex Learning , 2003 .

[111]  Sheila Lennon,et al.  Advances in Motor Learning and Control , 1997 .

[112]  Laila Dybkjær,et al.  Spoken Multimodal Human-Computer Dialogue in Mobile Environments , 2005 .

[113]  Robert N. Carson.  A Taxonomy of Knowledge Types for Use in Curriculum Design , 2004 .

[114]  Benjamin S. Bloom,et al.  Taxonomy of Educational Objectives: The Classification of Educational Goals. , 1957 .

[115]  Andrew E. Johnson,et al.  Designing cranial implants in a haptic augmented reality environment , 2004, CACM.

[116]  Gabriele Wulf,et al.  Self-Controlled Feedback: Does it Enhance Learning Because Performers Get Feedback When They Need It? , 2002, Research quarterly for exercise and sport.

[117]  H. Schlosberg,et al.  The retention of verbal and of motor skills , 1944 .

[118]  Eduard H. Hovy,et al.  Methodologies for the Reliable Construction of Ontological Knowledge , 2005, ICCS.

[119]  Anders Stenberg,et al.  Skill needs and continuing vocational training in Sweden , 2006 .

[120]  Timo Määttä,et al.  OSKU. Digital Human Model in the Participatory Design Approach: A New Tool to Improve Work Tasks and Workplaces , 2007 .

[121]  A. Bandura.  Social Foundations of Thought and Action: A Social Cognitive Theory , 1985 .

[122]  Jessie Borthwick,et al.  Evidence of Skill Shortages in the Automotive Repairs and Service Trades. , 2000 .

[123]  Robert J. Seidel,et al.  From Principles of Learning to Strategies for Instruction , 2006 .

[124]  Gobinda G. Chowdhury,et al.  Spinning the Semantic Web: Bringing the World Wide Web to Its Full Potential , 2004 .

[125]  M. Tuggy,et al.  Virtual Reality Flexible Sigmoidoscopy Simulator Training: Impact on Resident Performance , 1998, The Journal of the American Board of Family Medicine.

[126]  Benjamin S. Bloom,et al.  A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives , 2000 .

[127]  K. Newell.  Knowledge Of Results And Motor Learning , 1976, Exercise and sport sciences reviews.

[128]  Stephen Fox,et al.  Building information models, display media, and team performance , 2007 .

[129]  Hermann Bondi,et al.  The good, the bad and the ugly , 1988, Nature.

[130]  J. Stanley,et al.  Book Review: Taxonomy of Educational Objectives, The Classification of Educational Goals, Handbook I: Cognitive Domain , 1957 .

[131]  Alan Shipman,et al.  Skill Shortages in Britain and Germany , 1991 .

[132]  H. Knecht.  A Taxonomy of the Psychomotor Domain: A Guide for Developing Behavioral Objectives , 1974 .

[133]  Richard C. Anderson,et al.  Educational psychology;: The science of instruction and learning , 1973 .

[134]  Alan R. Dennis,et al.  Rethinking media richness: towards a theory of media synchronicity , 1999, Proceedings of the 32nd Annual Hawaii International Conference on Systems Sciences. 1999. HICSS-32. Abstracts and CD-ROM of Full Papers.

[135]  Christopher Martin,et al.  THE CAUSES OF SKILL SHORTAGES IN BRITAIN , 1993 .

[136]  Shamkant B. Navathe,et al.  Grammatical rules for specifying information for automated product data modeling , 2006, Adv. Eng. Informatics.

[137]  Debra T. Burhans,et al.  Visual Semantics: Extracting Visual information from Text Accompanying Pictures , 1994, AAAI.