Multimodal Data Streaming using Visual IoTs and Wearables for Telerehabilitation and Teletreatment

The current pandemic and the expansion of insurance coverage of telehealth for the elderly has shown the need for improvements in the automation of medical services and behavioral monitoring. Elderly individuals at home or in senior centers typically contact their physicians once a serious medical condition is obvious. There are very few systems with automated means of monitoring the health of individuals recovering from an injury or monitoring their progress when conducting telerehabilitation using wearables, remote cameras, and the cloud. To address these automation and long-term health monitoring issues, we have developed a multi-modal platform that employs common, low-cost wearables, and teleconferencing tools that takes advantage of cloud streaming services. One modality is based on Visual IoT and edge devices that automatically connect as teleconferencing clients to record human subjects’ telerehabilitation and teletreatment sessions and push these recordings to cloud storage. We performed experiments applying two different deep learning models to the cloud video streams to track whole-body skeleton motion or to detect hand keypoints during telerehabilitation. The rehabilitation maybe associated with recovery from a stroke or other similar incident. The other modality of our platform includes an Apple Watch ® App that pushes users’ physiological and kinematic data to the cloud and automatically generates data analytics dashboards for use by clinicians. Both, the wearables-based and the Visual IoT-based modalities complement each other in telerehabilitation or teletreatment controlled by clinicians. These modalities can also work together in sessions without the need for controlled (uncontrolled) telemonitoring. Our multimodality, cloud-based architecture shows very promising applications in automatic behavioral analytics, especially for elderly individuals with limited social contact that would benefit from telerehabilitation, or continuous telemonitoring.