Visual troubleshooting for robotic tasks

A robot system includes a robot, sensors which detect the status information including a position and orientation of the robot and an object within the working space, and a controller. The controller that examines an operation of the robot visually for error, includes a simulator module, an action planner module and a graphical user interface (GUI). The simulator module receives the status information and generates in response to marking instructions visual markers as graphical representations of the object and the robot. An action planner module selects a next action of the robot. The mark generator module generates the marking commands and outputs them in response to the selected next action on the simulator module. The GUI receives the visual markings, the chosen future action and input commands and displays them. By means of the action planner module, the position and / or orientation of the visual indicia be modified in real time in order to change the operation of the robot.