Sensor Architectures and Technologies for Upper Limb 3D Surface Reconstruction: A Review

3D digital models of the upper limb anatomy represent the starting point for the design process of bespoke devices, such as orthoses and prostheses, which can be modeled on the actual patient’s anatomy by using CAD (Computer Aided Design) tools. The ongoing research on optical scanning methodologies has allowed the development of technologies that allow the surface reconstruction of the upper limb anatomy through procedures characterized by minimum discomfort for the patient. However, the 3D optical scanning of upper limbs is a complex task that requires solving problematic aspects, such as the difficulty of keeping the hand in a stable position and the presence of artefacts due to involuntary movements. Scientific literature, indeed, investigated different approaches in this regard by either integrating commercial devices, to create customized sensor architectures, or by developing innovative 3D acquisition techniques. The present work is aimed at presenting an overview of the state of the art of optical technologies and sensor architectures for the surface acquisition of upper limb anatomies. The review analyzes the working principles at the basis of existing devices and proposes a categorization of the approaches based on handling, pre/post-processing effort, and potentialities in real-time scanning. An in-depth analysis of strengths and weaknesses of the approaches proposed by the research community is also provided to give valuable support in selecting the most appropriate solution for the specific application to be addressed.

[1]  Jonathan Kofman,et al.  Real-time 3D surface-shape measurement using background-modulated modified Fourier transform profilometry with geometry-constraint , 2019, Optics and Lasers in Engineering.

[2]  Jinwoong Kim,et al.  Development of Real-Time Hand Gesture Recognition for Tabletop Holographic Display Interaction Using Azure Kinect , 2020, Sensors.

[3]  ZhenZhou Wang,et al.  Single-shot three-dimensional reconstruction based on structured light line pattern , 2018, Optics and Lasers in Engineering.

[4]  Pedro Arias,et al.  Metrological comparison between Kinect I and Kinect II sensors , 2015 .

[5]  Monika Michalíková,et al.  Innovative approaches to designing and manufacturing a prosthetic thumb. , 2020, Prosthetics and orthotics international.

[6]  Sandro Barone,et al.  Low-frame-rate single camera system for 3D full-field high-frequency vibration measurements , 2019, Mechanical Systems and Signal Processing.

[7]  Carme Torras,et al.  ToF cameras for active vision in robotics , 2014 .

[8]  Xianyu Su,et al.  Fourier transform profilometry:: a review , 2001 .

[9]  Sandro Barone,et al.  Design and manufacturing of patient-specific orthodontic appliances by computer-aided engineering techniques , 2018, Proceedings of the Institution of Mechanical Engineers. Part H, Journal of engineering in medicine.

[10]  Yvon Voisin,et al.  Calibration of a three-dimensional reconstruction system using a structured light source , 2002 .

[11]  M. Kouchi Anthropometric methods for apparel design: Body measurement devices and techniques , 2014 .

[12]  Ping-Sing Tsai,et al.  Shape from Shading: A Survey , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[13]  Xin Li,et al.  Digital anthropometry: a critical review , 2018, European Journal of Clinical Nutrition.

[14]  D. F. Redaelli,et al.  LOW-COST 3D DEVICES AND LASER SCANNERS COMPARISON FOR THE APPLICATION IN ORTHOPEDIC CENTRES , 2018, The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences.

[15]  Andrea Tagliasacchi,et al.  Robust Articulated-ICP for Real-Time Hand Tracking , 2015 .

[16]  Lin Shi,et al.  A rapid and intelligent designing technique for patient-specific and 3D-printed orthopedic cast , 2016, 3D Printing in Medicine.

[17]  Hiroya Tanaka,et al.  Feasibility study applying a parametric model as the design generator for 3D–printed orthosis for fracture immobilization , 2018, 3D Printing in Medicine.

[18]  Alexandru Tupan,et al.  Triangulation , 1997, Comput. Vis. Image Underst..

[19]  Richard J. Bibb,et al.  Comparing additive manufacturing technologies for customised wrist splints , 2015 .

[20]  Dah-Jye Lee,et al.  Review of stereo vision algorithms and their suitability for resource-limited systems , 2013, Journal of Real-Time Image Processing.

[21]  Daniel J. Gould,et al.  Vectra 3D Imaging for Quantitative Volumetric Analysis of the Upper Limb: A Feasibility Study for Tracking Outcomes of Lymphedema Treatment , 2018, Plastic and reconstructive surgery.

[22]  S. Barone,et al.  Optical Stereo-System for Full-Field High-Frequency 3D Vibration Measurements Based on Low-Frame-Rate Cameras , 2019, Lecture Notes in Mechanical Engineering.

[23]  Khalil Khalili,et al.  A Fast and Low-Cost Human Body 3D Scanner Using 100 Cameras , 2020, J. Imaging.

[24]  Lapo Governi,et al.  A CAD-based Procedure for Designing 3D Printable Arm-Wrist-Hand Cast , 2018 .

[25]  Song Zhang,et al.  Trapezoidal phase-shifting method for 3D shape measurement , 2004, SPIE Optics East.

[26]  Zhenzhou Wang,et al.  Review of real-time three-dimensional shape measurement techniques , 2020 .

[27]  Song Zhang,et al.  High-speed 3D shape measurement with structured light methods: A review , 2018, Optics and Lasers in Engineering.

[28]  Pierre Grussenmeyer,et al.  Assessment of the accuracy of 3D models obtained with DSLR camera and Kinect v2 , 2015, Optical Metrology.

[29]  Alberto Signoroni,et al.  Deformable registration using patch-wise shape matching , 2014, Graph. Model..

[30]  Monica Carfagni,et al.  Fast and low cost acquisition and reconstruction system for human hand-wrist-arm anatomy , 2017 .

[31]  Andrew W. Fitzgibbon,et al.  KinectFusion: Real-time dense surface mapping and tracking , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[32]  Matthias Nießner,et al.  State of the Art on 3D Reconstruction with RGB‐D Cameras , 2018, Comput. Graph. Forum.

[33]  Pierre Graebling,et al.  Design of a Monochromatic Pattern for a Robust Structured Light Coding , 2007, 2007 IEEE International Conference on Image Processing.

[34]  Monica Carfagni,et al.  Metrological and Critical Characterization of the Intel D415 Stereo Depth Camera , 2018, Sensors.

[35]  Monica Carfagni,et al.  On the Performance of the Intel SR30 Depth Camera: Metrological and Critical Characterization , 2017, IEEE Sensors Journal.

[36]  Bing Pan,et al.  Digital image correlation for surface deformation measurement: historical developments, recent advances and future goals , 2018, Measurement Science and Technology.

[37]  Tokuo Tsuji,et al.  High-speed 3D image acquisition using coded structured light projection , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[38]  Marc Levoy,et al.  Real-time 3D model acquisition , 2002, ACM Trans. Graph..

[39]  Chuang-Yuan Chiu,et al.  Comparison of depth cameras for three-dimensional reconstruction in medicine , 2019, Proceedings of the Institution of Mechanical Engineers. Part H, Journal of engineering in medicine.

[40]  Brad E Dicianno,et al.  Innovations With 3‐Dimensional Printing in Physical Medicine and Rehabilitation: A Review of the Literature , 2016, PM & R : the journal of injury, function, and rehabilitation.

[41]  Monica L H Jones,et al.  Anthropometric Dimensions of Individuals With High Body Mass Index , 2019, Hum. Factors.

[42]  Yasushi Yagi,et al.  Dynamic scene shape reconstruction using a single structured light pattern , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[43]  Alberto Signoroni,et al.  3D scanning and geometry processing techniques for customised hand orthotics: an experimental assessment , 2018 .

[44]  Mitsuo Takeda,et al.  Fourier transform profilometry , 2000 .

[45]  Emanuele Zappa,et al.  Static and dynamic features of Fourier transform profilometry: A review , 2012 .

[46]  Stephen Charles Hsu,et al.  Performance of a Time-of-Flight Range Camera for Intelligent Vehicle Safety Applications , 2006 .

[47]  Sandro Barone,et al.  3D acquisition and stereo-camera calibration by active devices: A unique structured light encoding framework , 2020 .

[48]  Mircea Nicolescu,et al.  Vision-based hand pose estimation: A review , 2007, Comput. Vis. Image Underst..

[49]  Luc Van Gool,et al.  Real-time range acquisition by adaptive structured light , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[50]  T. Maal,et al.  Development of a three-dimensional hand model using 3D stereophotogrammetry: Evaluation of landmark reproducibility. , 2015, Journal of plastic, reconstructive & aesthetic surgery : JPRAS.

[51]  D. Meadows,et al.  Generation of surface contours by moiré patterns. , 1970, Applied optics.

[52]  A Agustsson,et al.  Validity and reliability of an iPad with a three-dimensional camera for posture imaging. , 2019, Gait & posture.

[53]  Christophe Collewet,et al.  Optimised De Bruijn patterns for one-shot shape acquisition , 2005, Image Vis. Comput..

[54]  Anand Asundi,et al.  Comparison of Fourier transform, windowed Fourier transform, and wavelet transform methods for phase extraction from a single fringe pattern in fringe projection profilometry , 2010 .

[55]  Song Zhang,et al.  Absolute phase retrieval methods for digital fringe projection profilometry: A review , 2018 .

[56]  Gabriel Taubin,et al.  One-shot scanning using De Bruijn spaced grids , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.

[57]  Bert Arnrich,et al.  Evaluation of the Pose Tracking Performance of the Azure Kinect and Kinect v2 for Gait Analysis in Comparison with a Gold Standard: A Pilot Study , 2020, Sensors.

[58]  David Paloušek,et al.  3D Digitalization of the Human Body for Use in Orthotics and Prosthetics , 2012 .

[59]  G. T. Reid Moiré fringes in metrology , 1984 .

[60]  Elise Lachat,et al.  Assessment and Calibration of a RGB-D Camera (Kinect v2 Sensor) Towards a Potential Use for Close-Range 3D Modeling , 2015, Remote. Sens..

[61]  Chih-Hsing Chu,et al.  Customized designs of short thumb orthoses using 3D hand parametric models , 2019, Assistive technology : the official journal of RESNA.

[62]  LevoyMarc,et al.  Real-time 3D model acquisition , 2002 .

[63]  Robert Sitnik,et al.  Four-dimensional measurement by a single-frame structured light method. , 2009, Applied optics.

[64]  Qian Chen,et al.  Phase shifting algorithms for fringe projection profilometry: A review , 2018, Optics and Lasers in Engineering.

[65]  S. Foix,et al.  Lock-in Time-of-Flight (ToF) Cameras: A Survey , 2011, IEEE Sensors Journal.

[66]  Alberto Signoroni,et al.  A Critical Analysis of a Hand Orthosis Reverse Engineering and 3D Printing Process , 2016, Applied bionics and biomechanics.

[67]  Jan Rosicky,et al.  Application of 3D Scanning in Prosthetic and Orthotic Clinical Practice , 2016 .

[68]  Caterina Rizzi,et al.  3D Scanning Procedure for the Evaluation of Lymphedema of Upper Limbs Using Low-Cost Technology: A Preliminary Study , 2019, DSMIE-2019.

[69]  Min Young Kim,et al.  Single shot laser speckle based 3D acquisition system for medical applications , 2018, Optics and Lasers in Engineering.

[70]  David P. Towers,et al.  Snapshot color fringe projection for absolute three-dimensional metrology of video sequences , 2010 .

[71]  Chad English,et al.  Error compensation in two-step triangular-pattern phase-shifting profilometry , 2008 .

[72]  Sam Van der Jeught,et al.  Real-time structured light profilometry: a review , 2016 .

[73]  HoraudRadu,et al.  An overview of depth cameras and range scanners based on time-of-flight technologies , 2016 .

[74]  Andrew W. Fitzgibbon,et al.  KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera , 2011, UIST.

[75]  Fu-Pen Chiang,et al.  High-speed 3-D shape measurement based on digital fringe projection , 2003 .

[76]  Marc Pollefeys,et al.  Capturing Hands in Action Using Discriminative Salient Points and Physics Simulation , 2015, International Journal of Computer Vision.

[77]  Zhenzhou Wang,et al.  Three-Dimensional Hand Reconstruction by Single-Shot Structured Light Line Pattern , 2018, IEEE Access.

[78]  H. A. M. Daanen,et al.  3D whole body scanners revisited , 2013, Displays.

[79]  Tomáš Návrat,et al.  Pilot study of the wrist orthosis design process , 2014 .

[80]  Richard W. Bohannon,et al.  Clinical measurement of range of motion. Review of goniometry emphasizing reliability and validity. , 1987, Physical therapy.

[81]  José A. Ferrari,et al.  One-shot 3D gradient field scanning , 2015 .

[82]  Ligang Liu,et al.  Scanning 3D Full Human Bodies Using Kinects , 2012, IEEE Transactions on Visualization and Computer Graphics.

[83]  Yi Yang,et al.  Depth-Based Hand Pose Estimation: Methods, Data, and Challenges , 2015, International Journal of Computer Vision.

[84]  Stanislao Grazioso,et al.  Design and development of a novel body scanning system for healthcare applications , 2018 .

[85]  Richard J. Bibb,et al.  A review of existing anatomical data capture methods to support the mass customisation of wrist splints , 2010 .

[86]  Fu-Pen Chiang,et al.  Color phase-shifting technique for three-dimensional shape measurement , 2006 .

[87]  Zhan Song,et al.  A single-shot structured light means by encoding both color and geometrical features , 2016, Pattern Recognit..

[88]  Xiaohui Zhou,et al.  Multiview phase shifting: a full-resolution and high-speed 3D measurement framework for arbitrary shape dynamic objects. , 2013, Optics letters.

[89]  Jingang Zhong,et al.  Phase retrieval of optical fringe patterns from the ridge of a wavelet transform. , 2005, Optics letters.

[90]  Huhn Kim,et al.  Case study: Hybrid model for the customized wrist orthosis using 3D printing , 2015 .

[91]  Iman Dianat,et al.  A review of the methodology and applications of anthropometry in ergonomics and product design , 2018, Ergonomics.

[92]  David Rodríguez Salgado,et al.  Advances in Orthotic and Prosthetic Manufacturing: A Technology Review , 2020, Materials.

[93]  Domenec Puig,et al.  Analysis of focus measure operators for shape-from-focus , 2013, Pattern Recognit..

[94]  Yangang Wang,et al.  Hand-3d-Studio: A New Multi-View System for 3d Hand Reconstruction , 2020, ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[95]  Charles Soussen,et al.  Flexible calibration of structured-light systems projecting point patterns , 2013, Comput. Vis. Image Underst..

[96]  Bilal Msallem,et al.  Three-dimensional Assessment of the Breast: Validation of a Novel, Simple and Inexpensive Scanning Process , 2019, In Vivo.

[97]  Cynthia L. Istook,et al.  Comparison of different body measurement techniques: 3D stationary scanner, 3D handheld scanner, and tape measurement , 2019 .

[98]  Rocco Furferi,et al.  A practical methodology for computer-aided design of custom 3D printable casts for wrist fractures , 2020, The Visual Computer.

[99]  Alan Lewis,et al.  Development of a 3D scan posture-correction procedure to facilitate the direct-digital splinting approach , 2018, Virtual and Physical Prototyping.

[100]  Qi Zhou,et al.  Three-dimensional reconstruction with single-shot structured light dot pattern and analytic solutions , 2020 .

[101]  Monica Bordegoni,et al.  A Wearable Device to Detect in Real-Time Bimanual Gestures of Basketball Players During Training Sessions , 2018, J. Comput. Inf. Sci. Eng..