Refraction corrected calibration for aquatic locomotion research: application of Snell’s law improves spatial accuracy

Images of underwater objects are distorted by refraction at the water-glass-air interfaces and these distortions can lead to substantial errors when reconstructing the objects' position and shape. So far, aquatic locomotion studies have minimized refraction in their experimental setups and used the direct linear transform algorithm (DLT) to reconstruct position information, which does not model refraction explicitly. Here we present a refraction corrected ray-tracing algorithm (RCRT) that reconstructs position information using Snell's law. We validated this reconstruction by calculating 3D reconstruction error-the difference between actual and reconstructed position of a marker. We found that reconstruction error is small (typically less than 1%). Compared with the DLT algorithm, the RCRT has overall lower reconstruction errors, especially outside the calibration volume, and errors are essentially insensitive to camera position and orientation and the number and position of the calibration points. To demonstrate the effectiveness of the RCRT, we tracked an anatomical marker on a seahorse recorded with four cameras to reconstruct the swimming trajectory for six different camera configurations. The RCRT algorithm is accurate and robust and it allows cameras to be oriented at large angles of incidence and facilitates the development of accurate tracking algorithms to quantify aquatic manoeuvers.

[1]  Adrian L. R. Thomas,et al.  Photogrammetric reconstruction of high-resolution surface topographies and deformable wing kinematics of tethered locusts and free-flying hoverflies , 2009, Journal of The Royal Society Interface.

[2]  Michael H Dickinson,et al.  Wing and body motion during flight initiation in Drosophila revealed by automated visual tracking , 2009, Journal of Experimental Biology.

[3]  G. Wood,et al.  The accuracy of DLT extrapolation in three-dimensional film analysis. , 1986, Journal of biomechanics.

[4]  Minglun Gong,et al.  Underwater Camera Calibration Using Wavelength Triangulation , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[5]  Gordon J. Berman,et al.  Automated hull reconstruction motion tracking (HRMT) applied to sideways maneuvers of free-flying insects , 2009, Journal of Experimental Biology.

[6]  Takashi Matsuyama,et al.  Underwater 3D Surface Capture Using Multi-view Projectors and Cameras with Flat Housings , 2014, IPSJ Trans. Comput. Vis. Appl..

[7]  Tamar Flash,et al.  Analyzing octopus movements using three-dimensional reconstruction. , 2007, Journal of neurophysiology.

[8]  Yee-Hong Yang,et al.  Two-View Camera Housing Parameters Calibration for Multi-layer Flat Refractive Interface , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[9]  G. Lauder,et al.  Functional morphology of the fin rays of teleost fishes , 2013, Journal of morphology.

[10]  S. N. Fry,et al.  The Aerodynamics of Free-Flight Maneuvers in Drosophila , 2003, Science.

[11]  M. A. Sutton,et al.  Robust stereo vision and calibration methodology for accurate three-dimensional digital image correlation measurements on submerged objects , 2008 .

[12]  James S. Walton,et al.  Underwater Tracking In Three Dimensions Using The Direct Linear Transformation And A Video-Based Motion Analysis System , 1988, Optics & Photonics.

[13]  Christopher Koehler,et al.  3D reconstruction and analysis of wing deformation in free-flying dragonflies , 2012, Journal of Experimental Biology.

[14]  Alison L. Sheets,et al.  Quantitative Evaluation of 3D Mouse Behaviors and Motor Function in the Open-Field after Spinal Cord Injury Using Markerless Motion Tracking , 2013, PloS one.

[15]  Jun Sakakibara,et al.  Stereo-PIV study of flow around a maneuvering fish , 2004 .

[16]  Jesse Belden,et al.  Calibration of multi-camera systems with refractive interfaces , 2013 .

[17]  Willmott,et al.  Measuring the angle of attack of beating insect wings: robust three-dimensional reconstruction from two-dimensional images , 1997, The Journal of experimental biology.

[18]  Arjan Kuijper,et al.  Converting underwater imaging into imaging in air , 2015, 2014 International Conference on Computer Vision Theory and Applications (VISAPP).

[19]  Anne Jordt,et al.  Underwater 3D Reconstruction Based on Physical Models for Refraction and Underwater Light Propagation , 2014 .

[20]  Takashi Matsuyama,et al.  3D Shape from Silhouettes in Water for Online Novel-view Synthesis , 2013, IPSJ Trans. Comput. Vis. Appl..

[21]  Kevin L. Woo,et al.  From dummies to animations: a review of computer-animated stimuli used in animal behavior studies , 2011, Behavioral Ecology and Sociobiology.

[22]  Tyson L Hedrick,et al.  Software techniques for two- and three-dimensional kinematic measurements of biological and biomimetic systems , 2008, Bioinspiration & biomimetics.

[23]  Margrit Betke,et al.  A protocol and calibration method for accurate multi-camera field videography , 2014, Journal of Experimental Biology.

[24]  Shahin Sefati,et al.  Mutually opposing forces during locomotion can eliminate the tradeoff between maneuverability and stability , 2013, Proceedings of the National Academy of Sciences.

[25]  J. L. Leeuwen,et al.  Undulatory fish swimming: from muscles to flow , 2006 .

[26]  H. Singh,et al.  Hemispherical refraction and camera calibration in underwater vision , 2008, OCEANS 2008.

[27]  Visesh Chari,et al.  A theory of multi-layer flat refractive geometry , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[28]  R. W. Austin,et al.  The Index of Refraction of Seawater , 1976 .

[29]  George V Lauder,et al.  Escaping Flatland: three-dimensional kinematics and hydrodynamics of median fins in fishes , 2008, Journal of Experimental Biology.

[30]  P. Perona,et al.  Visual methods for three-dimensional modeling , 1999 .

[31]  Yee-Hong Yang,et al.  Experimental study of the influence of refraction on underwater three-dimensional reconstruction using the SVP camera model. , 2012, Applied optics.

[32]  Hans-Peter Seidel,et al.  Free-viewpoint video of human actors , 2003, ACM Trans. Graph..

[33]  Tsuhan Chen,et al.  Multi-view 3D reconstruction for scenes under the refractive plane with known vertical direction , 2011, 2011 International Conference on Computer Vision.

[34]  L. Chen,et al.  An investigation on the accuracy of three-dimensional space reconstruction using the direct linear transformation technique. , 1994, Journal of biomechanics.

[35]  Ajay K. Prasad,et al.  Stereoscopic particle image velocimetry , 2000 .

[36]  D. B. Baier,et al.  X-ray reconstruction of moving morphology (XROMM): precision, accuracy and applications in comparative biomechanics research. , 2010, Journal of experimental zoology. Part A, Ecological genetics and physiology.

[37]  Young-Hoo Kwon,et al.  Effects of light refraction on the accuracy of camera calibration and reconstruction in underwater motion analysis , 2006, Sports biomechanics.

[38]  Hans-Peter Seidel,et al.  Performance capture from sparse multi-view video , 2008, SIGGRAPH 2008.

[39]  Anne Jordt,et al.  Calibration of Housing Parameters for Underwater Stereo-Camera Rigs , 2011, BMVC.

[40]  Sachit Butail,et al.  Three-dimensional reconstruction of the fast-start swimming kinematics of densely schooling fish , 2012, Journal of The Royal Society Interface.

[41]  Runan Huang,et al.  The Underwater Camera Calibration , 2014 .

[42]  H. Hatze,et al.  High-precision three-dimensional photogrammetric calibration and object space reconstruction using a modified DLT-approach. , 1988, Journal of biomechanics.

[43]  Meirav Galun,et al.  Nearly automatic motion capture system for tracking octopus arm movements in 3D space , 2009, Journal of Neuroscience Methods.

[44]  Antoine Manzanera,et al.  Automatic analysis and characterization of the hummingbird wings motion using dense optical flow features , 2015, Bioinspiration & biomimetics.

[45]  George V. Lauder,et al.  Swimming hydrodynamics: ten questions and the technical approaches needed to resolve them , 2011 .

[46]  Lai Kang,et al.  Theory of multi-level refractive geometry , 2015 .