PURPOSE:To obtain a satisfactory finished surface by incorporating a visual sensor so as to control a tool in accordance with removal shape data concerning a surface of a part to be removed and making direct contact with the tool during machining, when the tool is to be controlled in accordance with recognized data as to the removal shape of the workpiece. CONSTITUTION:In the case of application to a deburring process for a casting, surface shape data delivered from a visual sensor 2 attached to the hand tip of a robot 1 is a lower grade computer 5 for determining deburring parameters in accordance with a detection signal, and then the data is transmitted to a robot control device 8. Further, the robot 1 is maneuvered so as to deburr a workpiece 9 while control is made such that a predetermined pushing pressure can be obtained in accordance with a value delivered from a force sensor 3. In this phase, the robot is maneuvered in accordance with a maneuvering locus of the robot during visual recognition which is generated from CAD data of the workpiece 9, so as to carry out deburring shape recognition. Thus, a deburring schedule is prepared in accordance with the deburring shape data and the CAD data of the workpiece.