Development of Bridge Operation Support System using Multimodal Sensing

We have developed a multimodal sensing system to support team operation in ship’s bridge using sailor’s behavior data. We have implemented functions to detect various features including timing of utterances, speaker of each utterances, and sailor’s location. Comparing detected features with sailor’s workflow of bridge operation, the system enables to offer information like ratio of initiative utterance, amount of utterances while radio transmission. We have applied the system to data recorded in ship handling simulator, and the system have generated information which is useful for analyzing sailor’s behavior and skills automatically from audio and video data. The result have shown that the system is effective for analyzing sailor’s behavior to enhance sailor’s skill and knowledge transfer.