A human-robot interface using an interactive hand pointer that projects a mark in the real work space

A human-robot interface system is under development that takes into account the flexibility of the DigitalDesk approach. The prototype consists of a projector subsystem for information display and a real-time tracking vision subsystem to recognize the human's action. Two levels of interaction using a virtual operational panel and interactive image panel have been developed. This paper presents the third subsystem, the interactive hand pointer used for selecting objects or positions in the environment via the operator's hand gestures. The system visually tracks the operator's pointing hand and projects a mark at the indicated position using an LCD projector. Since the mark can be observed directly in the real work space without monitor displays or HMDs, correction of the indicated position by moving the hand is very easy for the operator. The system enables projection of a mark not only at a target plane with a known height but also to a plane with an unknown height. Experimental results of a pick-and-place task demonstrate the usefulness of the proposed system.