HydraDoctor: real-time liquids intake monitoring by collaborative sensing

Water has been widely acknowledged as an essential part of all living things. It is the fundamental necessity for all life's activities and most biochemical reactions in human body are executed in water. Therefore, the type and quantity of liquid intake everyday have a critical impact on individuals' health. In this paper, we demonstrate HydraDoctor, a real-time liquids intake monitoring system which is able to detect drinking activities, classify the categories of liquids and estimate the amount of intake. The system runs on multiple platforms including a smartwatch to detect the motion of hands and a smartglass to capture the images of mugs. A smartphone is also used as an edge computing platform and a remote server is designed for computationally intensive image processing. In HydraDoctor, multiple state-of-the-art machine learning techniques are applied: a Support Vector Machine (SVM)-based classifier is proposed to achieve accurate and efficient liquids intake monitoring, which is trained to detect the hand raising action. Both of them are well optimized to enable in-situ processing on smartwatch. To provide more robust and detailed monitoring, the smartglass is also incorporated and trigged to capture a short video clip in the front of the user when potential drinking activity is detected. The smartglass will send the video clip to the remote server via its companion smartphone and a Faster-RCNN is performed on the server to confirm the detected drinking activity and identify the type of intake liquid. According to our evaluation on the real-world experiments, HydraDoctor achieves very high accuracy both in drinking activity detection and types of liquids classification, whose accuracy is 85.64% and 84% respectively.