Supporting Complex Tasks Using Multiple Devices

Intelligent digital assistants help people perform simple tasks such as question answering, event scheduling, etc. Complex task assistance requires monitoring goals, intentions, and task state. Additionally, complex tasks may exceed the capabilities of one single device or application, giving rise to the need to support people as they engage with two or more devices (e.g., smartphones, smart wearable devices, tablets, and desktop or laptop computers) simultaneously. Using multiple connected devices to support tasks has now become a viable design option for digital assistance since many users already own multiple devices. In this paper, we introduce a framework for developing multiple device experiences (MDX), then demonstrate some of the early capabilities of the framework by means of the AskChef cooking assistant. AskChef allows people to use their existing mobile devices (e.g., tablets), together with existing headless smart speakers (e.g., the Amazon Echo) for rich step-by-step guidance and contextual question answering over the culinary recipe loaded in their mobile device’s web browser. We conducted a pilot study in which people engaged in cooking recipes with the AskChef MDX system. All participants reported finding the support to be helpful, and indicated a willingness to use the system in the future. Nevertheless, AskChef erred on occasion, affording an opportunity to study how people respond to errors in multi-device scenarios. We examine participants’ perceptions and actions, and outline some directions for future research.