Instant 3D design concept generation and visualization by real-time hand gesture recognition

3D conceptual design is a creative process. While the designers are in the inspirational mode, they want to quickly express the design ideas in 3D and visualize it in real time without obstacle. In an exploration of a new, natural and more user friendly interface to assist rapid 3D conceptual generation and visualization, a prototype system of hand gesture and motion capture based user interface has been proposed and implemented. In this paper, we present the framework and components of this real time 3D conceptual generation and visualization system. Hand gestures were designed with the consideration of ease to use and the suitability for real time continuous recognition. Rule-based intelligent menu/icons were designed which changes the prompts according to the design process and expected operations. Real time hand gesture recognition was realized by skeleton model based template matching, and the use of hidden Markov models (HMMs). The recognized hand gestures become command script in OpenSCAD environment, and the 3D design concept was instantly generated and displayed on the screen. A case study was conducted for the initial evaluation of the system.

[1]  Nianjun Liu,et al.  Understanding HMM training for video gesture recognition , 2004, 2004 IEEE Region 10 Conference TENCON 2004..

[2]  Mircea Nicolescu,et al.  Vision-based hand pose estimation: A review , 2007, Comput. Vis. Image Underst..

[3]  Jiajie Zhang,et al.  The Interaction of Internal and External Representations in a Problem Solving Task , 1990 .

[4]  Andrew Roberts,et al.  3-Draw: a tool for designing 3D shapes , 1991, IEEE Computer Graphics and Applications.

[5]  Biing-Hwang Juang,et al.  Fundamentals of speech recognition , 1993, Prentice Hall signal processing series.

[6]  Rajit Gadh,et al.  Multi-sensory user interface for a virtual-reality-based computer-aided design system , 1997, Comput. Aided Des..

[7]  Jovan Popović,et al.  Real-time hand-tracking with a color glove , 2009, SIGGRAPH 2009.

[8]  Larry J. Leifer,et al.  DesignSpace: a manual interaction environment for computer aided design , 1994, CHI '94.

[9]  J. Michael Moshell,et al.  A Two-Handed Interface for Object Manipulation in Virtual Environments , 1995, Presence: Teleoperators & Virtual Environments.

[10]  Alexander Zelinsky,et al.  Visual gesture interfaces for virtual environments , 2002, Interact. Comput..

[11]  David Radcliffe,et al.  Impact of CAD tools on creative problem solving in engineering design , 2009, Comput. Aided Des..

[12]  Ram D. Sriram,et al.  Standardized data exchange of CAD models with design intent , 2007, Comput. Aided Des..

[13]  Andrew Wilson,et al.  Data miming: inferring spatial object descriptions from human gesture , 2011, CHI.

[14]  Jinsheng Kang,et al.  USE OF 3D BODY MOTION TO FREEFORM SURFACE DESIGN , 2006 .

[15]  David K. Wright,et al.  A novel form design and CAD modelling approach , 2008, Comput. Ind..

[16]  Soonhung Han,et al.  A procedural method to exchange editable 3D data from a free-hand 2D sketch modeling system into 3D mechanical CAD systems , 2012, Comput. Aided Des..

[17]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[18]  Naai-Jung Shih,et al.  Gesture modeling for architectural design , 1996, Comput. Graph..

[19]  Jinsheng Kang,et al.  Use of Three-Dimensional Body Motion to Free-Form Surface Design , 2006 .

[20]  Vladimir Pavlovic,et al.  Visual Interpretation of Hand Gestures for Human-Computer Interaction: A Review , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[21]  Ian Gibson,et al.  Desktop virtual reality interface for computer aided conceptual design using geometric techniques , 2001 .

[22]  Ayoub Al-Hamadi,et al.  A Hidden Markov Model-Based Isolated and Meaningful Hand Gesture Recognition , 2008 .

[23]  James D. Hollan,et al.  Distributed cognition: toward a new foundation for human-computer interaction research , 2000, TCHI.

[24]  Michael Girard,et al.  Computer animation of knowledge-based human grasping , 1991, SIGGRAPH.

[25]  Jin-Hyung Kim,et al.  An HMM-Based Threshold Model Approach for Gesture Recognition , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[26]  Karthik Ramani,et al.  Shape-It-Up: Hand gesture based creative expression of 3D shapes using intelligent generalized cylinders , 2013, Comput. Aided Des..

[27]  Ying Wu,et al.  Vision-Based Gesture Recognition: A Review , 1999, Gesture Workshop.

[28]  Joaquim A. Jorge,et al.  Mockup Builder: 3D modeling on and above the surface , 2013, Comput. Graph..

[29]  Tushar H. Dani,et al.  Evaluation of virtual reality interface for product shape designs , 1998 .

[30]  Jinsheng Kang,et al.  Generating 3D architectural models based on hand motion and gesture , 2009, Comput. Ind..

[31]  Tosiyasu L. Kunii,et al.  Model-based analysis of hand posture , 1995, IEEE Computer Graphics and Applications.

[32]  Alain Bernard,et al.  Virtual hands and virtual reality multimodal platform to design safer industrial systems , 2007, Comput. Ind..

[33]  Joaquim A. Jorge,et al.  Mockup builder: direct 3D modeling on and above the surface in a continuous interaction space , 2012, Graphics Interface.

[34]  Levent Burak Kara,et al.  Conceptual design and modification of freeform surfaces using dual shape representations in augmented reality environments , 2012, Comput. Aided Des..

[35]  José Teixeira,et al.  Advanced interaction techniques in virtual environments , 1993, Comput. Graph..

[36]  David K. Wright,et al.  Three-dimensional immersive mixed-reality interface for structural design , 2012 .

[37]  Yu Chen,et al.  Sketching in the air: A vision-based system for 3D object design , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[38]  S. Mitra,et al.  Gesture Recognition: A Survey , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[39]  George Moore,et al.  HandPuppet3D: Motion capture and analysis for character animation , 2009, Artificial Intelligence Review.