Screen Intelligence Engine: ML based method for predicting IoT devices using screen contents

Growing number of IoT devices and their varied capabilities pose problems for consumers like keeping efficient track of these devices and effectively utilize them. Screen Intelligence Engine (SIE) was conceived to enable users performing an activity on a device to enable another activity / task on a related IoT device seamlessly. SIE is based on the idea that the content and context of user activity on a device is one of the key factors in determining an action they would take on different IoT devices. The challenge of SIE is to understand the user intent from the content a user is consuming. In this paper, we propose a novel solution that uses text data available on a user’s device along with the application type context to classify and predict relevant IoT devices, making them available to the user in an instant. The proposed solution has achieved over 90% accuracy in predicting the relevant IoT devices. This will not only provide a seamless experience of IoT devices to users but also promote the growth and adoption of IoT based consumer electronics and home automation.