CONSTRUCTION OF A SOMATOSENSORY INTERACTIVE SYSTEM BASED ON COMPUTER VISION AND AUGMENTED REALITY TECHNIQUES USING THE KINECT DEVICE FOR FOOD AND AGRICULTURAL EDUCATION

A somatosensory interactive system based on computer vision and augmented reality (AR) techniques using the Kinect device is proposed, on which a game of harvesting three kinds of fruit can be played for food and agricultural education. The Kinect is used to capture users’ motion images, the Unity3D is used as the game engine, and the Kinect SDK is used for developing programs, to implement the tasks of face detection and tracking, hand-gesture recognition, and body-model matching and tracking involved in fruit-harvesting activities. AR-based photos of the harvest result can be taken and downloaded as souvenirs. The system was exhibited and observations of the users’ performances as well as interviews with experts and the users were conducted. The collected opinions were used to evaluate the effectiveness of the system, reaching the following conclusions: 1) the interactive experience of using this system is simple and intuitive; 2) the use of body movements for man-machine interaction is given positive reviews; 3) the introduction of somatosensory interactive education can arouse participants’ interest, achieving the effect of edutainment; and 4) the experience of taking commemorative photos can achieve the publicity and promotion effect of food and agricultural education through sharing on social media.

[1]  James W. Davis,et al.  The KidsRoom: A Perceptually-Based Interactive and Immersive Story Environment , 1999, Presence.

[2]  J. H. Gilmore,et al.  Welcome to the experience economy. , 1998, Harvard business review.

[3]  Aya H. Kimura.  Food education as food literacy: privatized and gendered food knowledge in contemporary Japan , 2011 .

[4]  C. Petrini,et al.  Slow Food Nation: Why Our Food Should Be Good, Clean, and Fair , 2007 .

[5]  Hui-mei Justina Hsu.  The Potential of Kinect in Education , 2011 .

[6]  Roger K. Moore,et al.  Handbook of Multimodal and Spoken Dialogue Systems: Resources, Terminology and Product Evaluation , 2000 .

[7]  Richard Szeliski,et al.  Computer Vision - Algorithms and Applications , 2011, Texts in Computer Science.

[8]  Sachin Sakhare,et al.  Image processing techniques for object tracking in video surveillance- A survey , 2015, 2015 International Conference on Pervasive Computing (ICPC).

[9]  David M. Agnew,et al.  Agricultural Literacy: Clarifying a Vision for Practical Application. , 2008 .

[10]  Hyunggi Kim,et al.  A study on the media arts using interactive projection mapping , 2014 .

[11]  Ronald Poppe,et al.  Vision-based human motion analysis: An overview , 2007, Comput. Vis. Image Underst..

[12]  Taysheng Jeng,et al.  INTERACTION AND SOCIAL ISSUES IN A HUMAN-CENTERED REACTIVE ENVIRONMENT , 2002 .

[13]  Alan L Eliason.  Systems Development: Analysis, Design, and Implementation , 1987 .

[14]  H. Pluckrose,et al.  Things to See , 1973 .

[15]  Marie-Francine Moens,et al.  Handbook of Multimodal and Spoken Dialogue Systems: Resources, Terminology and Product Evaluation , 2000, Computational Linguistics.

[16]  Nicu Sebe,et al.  Multimodal Human Computer Interaction: A Survey , 2005, ICCV-HCI.

[17]  James L. Crowley,et al.  Vision for man machine interaction , 1995 .

[18]  M. Turk,et al.  Perceptual Interfaces , 2003 .

[19]  Barry H. Kantowitz,et al.  Human Factors: Understanding People-System Relationships , 1983 .

[20]  Andoni Beristain Iraola.  Skeleton based visual pattern recognition: applications to tabletop interaction , 2009 .

[21]  Vladimir Pavlovic,et al.  Toward multimodal human-computer interface , 1998, Proc. IEEE.

[22]  PARplatform foragrobiodiversityresearch.  Biodiversity for Food and Agriculture Contributing to food security and sustainability in a changing world , 2011 .

[23]  HiltonAdrian,et al.  A survey of advances in vision-based human motion capture and analysis , 2006 .

[24]  Justus D. Naumann,et al.  Prototyping: the new paradigm for systems development , 1982 .

[25]  James L. Crowley,et al.  Perceptual user interfaces: things that see , 2000, CACM.

[26]  Sarah Diefenbach,et al.  Needs, affect, and interactive products - Facets of user experience , 2010, Interact. Comput..

[27]  Yvonne Rogers,et al.  Interaction Design: Beyond Human-Computer Interaction , 2002 .

[28]  R. S. Jadon,et al.  Computer Vision Based Human Computer Interaction , 2011 .

[29]  Golan Levin.  Computer vision for artists and designers: pedagogic tools and techniques for novice programmers , 2006, AI & SOCIETY.

[30]  Tom Geller,et al.  Interactive Tabletop Exhibits in Museums and Galleries , 2006, IEEE Computer Graphics and Applications.

[31]  A Tale of Technology and Collaboration: Preparing for 21st-Century Museum Visitors , 2019, Journal of Museum Education.

[32]  S. Malik,et al.  Use of Multimedia as a New Educational Technology Tool-A Study , 2012 .

[33]  L. Jackson,et al.  Special Issue Article: Advancing Environmental Conservation: Essays In Honor Of Navjot Sodhi Global food security, biodiversity conservation and the future of agricultural intensification , 2012 .

[34]  Andrew W. Fitzgibbon,et al.  Real-time human pose recognition in parts from single depth images , 2011, CVPR 2011.

[35]  Nicu Sebe,et al.  Multimodal Human Computer Interaction: A Survey , 2005, ICCV-HCI.

[36]  Austin Henderson,et al.  Interaction design: beyond human-computer interaction , 2002, UBIQ.

[37]  Suihuai Yu,et al.  Layout Design of Human-Machine Interaction Interface of Cabin Based on Cognitive Ergonomics and GA-ACA , 2016, Comput. Intell. Neurosci..

[38]  Tieniu Tan,et al.  A survey on visual surveillance of object motion and behaviors , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[39]  Mathias Kölsch,et al.  Emerging Topics in Computer Vision , 2004 .

[40]  Triangle Model Theory for Enhance the Usability by User Centered Design Process in Human Computer Interaction , 2017 .

[41]  Status of World Food Security and its Future Outlook, and Role of Agricultural Research and Education , 2016 .

[42]  Andrew D. Wilson.  PlayAnywhere: a compact interactive tabletop projection-vision system , 2005, UIST.

[43]  Adrian Hilton,et al.  A survey of advances in vision-based human motion capture and analysis , 2006, Comput. Vis. Image Underst..

[44]  William Lidwell,et al.  Universal principles of design : 100 ways to enhance usability,influence perception, increase appeal, make better, designdecisions, and teach through design , 2003 .

[45]  Kirubaraj Ragland,et al.  A Survey on Object Detection, Classification and Tracking Methods , 2014 .

[46]  Matthew Turk,et al.  Computer vision in the interface , 2004, CACM.

[47]  Jane Fulton Suri,et al.  Experience prototyping , 2000, DIS '00.