A trifocal tensor based camera-projector system for robot-human interaction

In this paper we present a novel real time camera-projector system for assisting robot-human communication using natural interfaces. The system is comprised of a stereo camera pair and a DLP projector. The proposed system provides feedback information indicating the robot's understanding of the environment and what action a human user desires. Feedback is delivered by iteratively spotlighting objects in the environment using a projector. When the spotlight lands on an object that the human user wants the robot to retrieve, he or she can confirm the object selection, and the robot will move to a position suitable for grasping the object. In this investigation, the proposed camera-projector performs three tasks: 1) Locate salient objects in the scene using a visual attention model and a trifocal tensor based object matching method. 2) Project a spot of light on detected objects to provide information back to the human user. 3) Reconstruct the 3D position of the target to plan robot motion and conduct vision-based control to move a robot manipulator to the object. A calibration procedure for three view camera-projector system based on the trifocal tensor is presented. Experimental results of human-directed visual servoing task are presented to demonstrate the functions of the proposed system.