Assembly Motion Recognition Framework Using Only Images

This work proposes a method for recognizing and segmenting assembly tasks into single motions. First, using a motion capture system based on pose estimation from multiple points, we obtain a time series data of the human’s motion during an assembly task (motion data). We use an object detector algorithm to determine the assembly parts and tools that the user (human) is grasping. Then, we divide (segment) the assembly motion based on the change of the manipulated object and the velocity of the hand. We carry out the motion recognition of the segmented motion data by using several Hidden Markov Models (HMMs) that represent the actions that can be executed with the manipulated object(s). We recorded the assembly motion of an airplane toy done by two experts for training the HMMs and recorded the assembly motion of five subjects to verify the validity of the proposed method.