A multi-camera system for underwater real-time 3D fish detection and tracking

Analyzing fish and fish schools behavior can help in studying fish-fish interaction, analyzing characteristics of fish species, studying prey avoidance maneuvers of fish schools, etc. Such analysis requires the estimation of each fish's 3D location, 3D pose, and 3D shape over time. Moreover if we are interested in studying the interaction of fish by injecting visual / acoustic stimuli artificially according to their motions, this information is required in real-time. In this context, our goal is to track in 3D each fish location and pose, accurately, in real-time, and in the future we foresee the use of underwater vehicles with multiple cameras for 3D fish school behavior analysis. As a step in this direction, in the current paper we propose the use of a calibrated multi-camera system, where each camera captures images through a flat surface, and the cameras observe a common region from different point of views (through one or more flat surfaces). The proposed system allows to detect and track in 3D each fish location in real-time, while taking into account light refraction though flat surfaces. We test the proposed approach using a fish tank with flat surfaces and present validation results and obtained processing times.