Semantic constraints to represent common sense required in household actions for multi-modal Learning-from-observation robot

Recent years have witnessed an increasing demand of service robots that can assist the elderly. Many elderlies now reside in senior residences that face support staff shortages. Further, although their residences are comfortable and well supported, the elderly still desire to live in their own homes when possible. To fulfill such requirements of living in their own homes necessitate require further support. Therefore, it is important and imminent to develop service robots that support the lives of the elderly in senior residences and/or in their own homes to meet these requirements. The paradigm of learning-from-observation (LfO) is a promising direction toward this goal. An LfO system observes human actions and learns how to perform these actions via the observations. In senior residences and senior homes, we assume that nurses and care workers are stationed or visiting on a part-time basis. These novice users can teach and/or modify the robot’s action through the LfO system instead of manual programming. Further, even though each home has a large variation in the environment, if necessary, such care takers can tune up the robot actions through their demonstrations on-site to absorb the environmental variations. Thus far, most LfO systems have been developed for relatively clean environments such as machine assembly in industrial settings as done by [1] or rope handling in laboratory settings as done by [2]. The home environment is cluttered and household actions have wide variations that require common sense to understand and pursuit the actions. To overcome this clut-