Fish motion capture with refraction synthesis

3D fish animations become more and more popular in fish behavioral research. It empowers the experimenter to design fish stimuli and their specific behavior to the experiment’s needs. The fish animation can be done manually or derived from video footage. Especially automatic fish model parameter recovery for 3D animations is not well studied yet. Here we present a novel, flexible method for this purpose. It can be used to recover position, pose, bone rotation and size from single or multiple view and for single or multiple fish. Additionally we implement a novel method to compensate the fish tank’s refraction effect and show that this method can decrease the error up to 80 %. We successfully applied the proposed method to two different data sets and recovered fish parameters out of singleand double-view video stream. A video attached to this paper demonstrates the results.

[1]  Klaus-Dieter Kuhnert,et al.  Fish motion simulation , 2015 .

[2]  Sachit Butail,et al.  Three-dimensional reconstruction of the fast-start swimming kinematics of densely schooling fish , 2012, Journal of The Royal Society Interface.

[3]  Gil G. Rosenthal,et al.  anyFish: an open-source software to generate animated fish models for behavioural studies , 2013 .

[4]  Sachit Butail,et al.  3D reconstruction of fish schooling kinematics from underwater video , 2010, 2010 IEEE International Conference on Robotics and Automation.

[5]  Klaus-Dieter Kuhnert,et al.  Optimal Feature-set Selection Controlled by Pose-space Location , 2016, VISIGRAPP.

[6]  Lars Kuhnert,et al.  Calibration and 3D ground truth data generation with orthogonal camera-setup and refraction compensation for aquaria in real-time , 2014, 2014 International Conference on Computer Vision Theory and Applications (VISAPP).

[7]  中嶋 正之,et al.  Animation Synthesis for Virtual Fish from Video , 2000 .

[8]  Pascal Poncin,et al.  Video multitracking of fish behaviour: a synthesis and future perspectives , 2013 .

[9]  John E. Dennis,et al.  An Adaptive Nonlinear Least-Squares Algorithm , 1977, TOMS.

[10]  Klaus-Dieter Kuhnert,et al.  The virtual lover: variable and easily guided 3D fish animations as an innovative tool in mate-choice experiments with sailfin mollies-I. Design and implementation , 2016, Current zoology.

[11]  Cees J. Voesenek,et al.  Automated Reconstruction of Three-Dimensional Fish Motion, Forces, and Torques , 2016, PloS one.

[12]  Yaser Sheikh,et al.  OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Varun Ramakrishna,et al.  Convolutional Pose Machines , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Andrew W. Fitzgibbon,et al.  Real-time human pose recognition in parts from single depth images , 2011, CVPR 2011.

[15]  Laura Chouinard-Thuly,et al.  Virtual is the new reality , 2017, Current zoology.

[16]  Mario Costa Sousa,et al.  The edge buffer: a data structure for easy silhouette rendering , 2000, NPAR '00.

[17]  Ronald Poppe,et al.  Vision-based human motion analysis: An overview , 2007, Comput. Vis. Image Underst..

[18]  Cristian Sminchisescu,et al.  Human Pose Estimation from Silhouettes - A Consistent Approach Using Distance Level Sets , 2002, WSCG.

[19]  D. Roetenberg,et al.  Xsens MVN: Full 6DOF Human Motion Tracking Using Miniature Inertial Sensors , 2009 .

[20]  Gil G. Rosenthal,et al.  anyFish 2.0: An open-source software platform to generate and share animated fish models to study behavior , 2015 .

[21]  Klaus-Dieter Kuhnert,et al.  The virtual lover: variable and easily guided 3D fish animations as an innovative tool in mate-choice experiments with sailfin mollies-II. Validation , 2016, Current zoology.