Telelife: The Future of Remote Living

In recent years, everyday activities such as work and socialization have steadily shifted to more remote and virtual settings. With the COVID-19 pandemic, the switch from physical to virtual has been accelerated, which has substantially affected almost all aspects of our lives, including business, education, commerce, healthcare, and personal life. This rapid and large-scale switch from in-person to remote interactions has exacerbated the fact that our current technologies lack functionality and are limited in their ability to recreate interpersonal interactions. To help address these limitations in the future, we introduce “Telelife,” a vision for the near and far future that depicts the potential means to improve remote living and better align it with how we interact, live and work in the physical world. Telelife encompasses novel synergies of technologies and concepts such as digital twins, virtual/physical rapid prototyping, and attention and context-aware user interfaces with innovative hardware that can support ultrarealistic graphics and haptic feedback, user state detection, and more. These ideas will guide the transformation of our daily lives and routines soon, targeting the year 2035. In addition, we identify opportunities across high-impact applications in domains related to this vision of Telelife. Along with a recent survey of relevant fields such as human-computer interaction, pervasive computing, and virtual reality, we provide a meta-synthesis in this paper that will guide future research on remote living.

[1]  Daniel Afergan,et al.  Dynamic difficulty using brain metrics of workload , 2014, CHI.

[2]  Katerina Mania,et al.  Near‐Eye Display and Tracking Technologies for Virtual and Augmented Reality , 2019, Comput. Graph. Forum.

[3]  Arzu Çöltekin,et al.  A Testbed Combining Visual Perception Models for Geographic Gaze Contingent Displays , 2015, EuroVis.

[4]  E. Biddiss,et al.  Active video game play in children with cerebral palsy: potential for physical activity promotion and rehabilitation therapies. , 2012, Archives of physical medicine and rehabilitation.

[5]  Pattie Maes,et al.  AttentivU: Designing EEG and EOG Compatible Glasses for Physiological Sensing and Feedback in the Car , 2019, AutomotiveUI.

[6]  Philipp A. Rauschnabel,et al.  Augmented reality smart glasses: an investigation of technology acceptance drivers , 2016 .

[7]  Robert Richer,et al.  Promoting relaxation using virtual reality, olfactory interfaces and wearable EEG , 2018, 2018 IEEE 15th International Conference on Wearable and Implantable Body Sensor Networks (BSN).

[8]  Marcus A. Magnor,et al.  Gaze-Contingent Computational Displays: Boosting perceptual fidelity , 2016, IEEE Signal Processing Magazine.

[9]  Charles T. Loop,et al.  Holoportation: Virtual 3D Teleportation in Real-time , 2016, UIST.

[10]  Alexander Brem,et al.  Who will buy smart glasses? Empirical results of two pre-market-entry studies on the role of personality in individual awareness and intended adoption of Google Glass wearables , 2015, Comput. Hum. Behav..

[11]  Philipp A. Rauschnabel,et al.  Understanding the diffusion of virtual reality glasses: The role of media, fashion and technology , 2019, Technological Forecasting and Social Change.

[12]  Thorsten O. Zander,et al.  Enhancing Human-Computer Interaction with Input from Active and Passive Brain-Computer Interfaces , 2010, Brain-Computer Interfaces.

[13]  Henry Fuchs,et al.  Improved vergence and accommodation via Purkinje Image tracking with multiple cameras for AR glasses , 2020, 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[14]  Dhruv Jain,et al.  Immersive Terrestrial Scuba Diving Using Virtual Reality , 2016, CHI Extended Abstracts.

[15]  Samuel B. Williams,et al.  ASSOCIATION FOR COMPUTING MACHINERY , 2000 .

[16]  Herman van der Auweraer,et al.  Digital Twins , 2020, SEMA SIMAI Springer Series.

[17]  Ronald Azuma,et al.  A Survey of Augmented Reality , 1997, Presence: Teleoperators & Virtual Environments.

[18]  Pierre Dillenbourg,et al.  Translating Head Motion into Attention - Towards Processing of Student's Body-Language , 2015, EDM.

[19]  JoAnn Difede,et al.  Development and early evaluation of the Virtual Iraq/Afghanistan exposure therapy system for combat‐related PTSD , 2010, Annals of the New York Academy of Sciences.

[20]  Sriram Subramanian,et al.  G-ID: Identifying 3D Prints Using Slicing Parameters , 2020, CHI.

[21]  Niels Henze,et al.  EngageMeter: A System for Implicit Audience Engagement Sensing Using Electroencephalography , 2017, CHI.

[22]  Marcus A. Magnor,et al.  An Affordable Solution for Binocular Eye Tracking and Calibration in Head-mounted Displays , 2015, ACM Multimedia.

[23]  Yuta Itoh,et al.  Beaming Displays , 2021, IEEE Transactions on Visualization and Computer Graphics.

[24]  Seungyeop Han,et al.  Exploring Cyberbullying and Other Toxic Behavior in Team Competition Online Games , 2015, CHI.

[25]  T. Ward,et al.  Brain computer interfaces for neurorehabilitation – its current status as a rehabilitation strategy post-stroke. , 2015, Annals of physical and rehabilitation medicine.

[26]  R. Parasuraman,et al.  Psychophysiology and adaptive automation , 1996, Biological Psychology.

[27]  Tobias Isenberg,et al.  Collaborative Work in Augmented Reality: A Survey , 2020, IEEE Transactions on Visualization and Computer Graphics.

[28]  Pattie Maes,et al.  AttentivU: A Wearable Pair of EEG and EOG Glasses for Real-Time Physiological Processing , 2019, 2019 IEEE 16th International Conference on Wearable and Implantable Body Sensor Networks (BSN).

[29]  Wendy Ju,et al.  Next Steps for Human-Computer Integration , 2020, CHI.

[30]  Homei Miyashita,et al.  Norimaki Synthesizer: Taste Display Using Ion Electrophoresis in Five Gels , 2020, CHI Extended Abstracts.

[31]  Yomna Abdelrahman,et al.  Exploring the Potential of Augmented Reality in Domestic Environments , 2019, MobileHCI.

[32]  Gordon Wetzstein,et al.  Varifocal Occlusion-Capable Optical See-through Augmented Reality Display based on Focus-tunable Optics , 2019, IEEE Transactions on Visualization and Computer Graphics.

[33]  Gregory D. Abowd,et al.  What next, ubicomp?: celebrating an intellectual disappearing act , 2012, UbiComp.

[34]  Wolfram Boucsein,et al.  Combining Skin Conductance and Heart Rate Variability for Adaptive Automation During Simulated IFR Flight , 2007, HCI.

[35]  David B. Kaber,et al.  Telepresence , 1998, Hum. Factors.

[36]  Kaan Akşit,et al.  Patch scanning displays: spatiotemporal enhancement for displays. , 2020, Optics express.

[37]  Kire Trivodaliev,et al.  A review of Internet of Things for smart home: Challenges and solutions , 2017 .

[38]  Didier Stricker,et al.  A Survey on Synchronous Augmented, Virtual, andMixed Reality Remote Collaboration Systems , 2021, ACM Comput. Surv..

[39]  Shwetak N. Patel,et al.  EyeContact: scleral coil eye tracking for virtual reality , 2016, SEMWEB.

[40]  Elizabeth F. Churchill,et al.  Collaborative virtual environments: An introductory review of issues and systems , 1998, Virtual Reality.

[41]  Maxim Shusteff,et al.  Volumetric additive manufacturing via tomographic reconstruction , 2019, Science.

[42]  Mark Billinghurst,et al.  Transitional interface: concept, issues and framework , 2006, 2006 IEEE/ACM International Symposium on Mixed and Augmented Reality.

[43]  Rongrong Fu,et al.  Dynamic driver fatigue detection using hidden Markov model in real driving condition , 2016, Expert Syst. Appl..

[44]  Barney Dalgarno,et al.  What are the learning affordances of 3-D virtual environments? , 2010, Br. J. Educ. Technol..

[45]  Teijo Lehtonen,et al.  Augmented reality - Towards an ethical fantasy? , 2014, 2014 IEEE International Symposium on Ethics in Science, Technology and Engineering.

[46]  WetzsteinGordon,et al.  Neural holography with camera-in-the-loop training , 2020 .

[47]  Franck Tarpin-Bernard,et al.  Towards Brain Computer Interfaces for Recreational Activities: Piloting a Drone , 2015, INTERACT.

[48]  R Kaviyaraj.,et al.  A Survey on Future of Augmented Reality with AI in Education , 2021, 2021 International Conference on Artificial Intelligence and Smart Systems (ICAIS).

[49]  Gordon Wetzstein,et al.  Event-Based Near-Eye Gaze Tracking Beyond 10,000 Hz , 2020, IEEE Transactions on Visualization and Computer Graphics.

[50]  G. Ryan "Your place or mine". , 1982, Obstetrics and Gynecology.

[51]  Ioannis Kazanidis,et al.  A systematic literature review of mixed reality environments in K-12 education , 2019, Education and Information Technologies.

[52]  Oleg V. Komogortsev,et al.  Can we beat the mouse with MAGIC? , 2013, CHI.

[53]  Christos Mousas,et al.  Human–virtual character interaction: Toward understanding the influence of haptic feedback , 2019, Comput. Animat. Virtual Worlds.

[54]  Anthony Steed,et al.  Networked Graphics - Building Networked Games and Virtual Environments , 2009 .

[55]  Chao Ping Chen,et al.  Design of foveated contact lens display for augmented reality. , 2019, Optics express.

[56]  Pattie Maes,et al.  AttentivU: An EEG-Based Closed-Loop Biofeedback System for Real-Time Monitoring and Improvement of Engagement for Personalized Learning , 2019, Sensors.

[57]  Sidney K. D'Mello,et al.  "Out of the Fr-Eye-ing Pan": Towards Gaze-Based Models of Attention during Learning with Technology in the Classroom , 2017, UMAP.

[59]  Florian Alt,et al.  Brainatwork: logging cognitive engagement and tasks in the workplace using electroencephalography , 2017, MUM.

[60]  S. Olesen,et al.  Technology and History , 2013 .

[61]  Zhong Fan,et al.  Digital Twin: Enabling Technologies, Challenges and Open Research , 2020, IEEE Access.

[62]  Alessandro Ricci,et al.  The Mirror World: Preparing for Mixed-Reality Living , 2015, IEEE Pervasive Computing.

[63]  Melvin Kranzberg Technology and History: "Kranzberg's Laws" , 1986 .

[64]  Clement Zheng,et al.  Compositional 3D printing: expanding & supporting workflows towards continuous fabrication , 2018, SCF.

[65]  Verena Fuchsberger,et al.  The future's hybrid nature , 2019, Interactions.

[66]  Holger Regenbrecht,et al.  ChromaGlasses: Computational Glasses for Compensating Colour Blindness , 2018, CHI.

[67]  Alexander Plopski,et al.  OrthoGaze: Gaze-based three-dimensional object manipulation using orthogonal planes , 2020, Comput. Graph..

[68]  Mark Weiser,et al.  The computer for the 21st Century , 1991, IEEE Pervasive Computing.

[69]  Gregory D. Abowd,et al.  Towards a Better Understanding of Context and Context-Awareness , 1999, HUC.

[70]  Andrei Ciortea,et al.  Hypermedia to connect them all: Autonomous hypermedia agents and socio‐technical interactions , 2018, Internet Technol. Lett..

[71]  Holger Regenbrecht,et al.  Towards Pervasive Augmented Reality: Context-Awareness in Augmented Reality , 2017, IEEE Transactions on Visualization and Computer Graphics.

[72]  D. Hamilton,et al.  Immersive virtual reality as a pedagogical tool in education: a systematic literature review of quantitative learning outcomes and experimental design , 2020, Journal of Computers in Education.

[73]  Richard H. R. Hahnloser,et al.  Eye-Trace: Segmentation of Volumetric Microscopy Images with Eyegaze , 2016, CHI.

[74]  Henry Fuchs,et al.  FocusAR: Auto-focus Augmented Reality Eyeglasses for both Real World and Virtual Imagery , 2018, IEEE Transactions on Visualization and Computer Graphics.

[75]  James Paul Gee,et al.  What video games have to teach us about learning and literacy , 2007, CIE.

[76]  Mashhuda Glencross,et al.  Cyber Security Threats and Challenges in Collaborative Mixed-Reality , 2019, Front. ICT.

[77]  L. Montgomery,et al.  Rheoencephalographic and electroencephalographic measures of cognitive workload: analytical procedures , 1995, Biological Psychology.

[78]  Patrick Baudisch,et al.  Scotty: Relocating Physical Objects Across Distances Using Destructive Scanning, Encryption, and 3D Printing , 2015, TEI.

[79]  Damla Turgut,et al.  A Systematic Review of the Convergence of Augmented Reality, Intelligent Virtual Agents, and the Internet of Things , 2019, Artificial Intelligence in IoT.

[80]  Robert W. Lindeman,et al.  Eye-tracking for human-centered mixed reality: promises and challenges , 2020, AR, VR, MR.

[81]  C. Mateer,et al.  Effectiveness of an attention-training program. , 1987, Journal of clinical and experimental neuropsychology.

[82]  Ömer Nezih Gerek,et al.  A Low-Computational Approach on Gaze Estimation With Eye Touch System , 2014, IEEE Transactions on Cybernetics.

[83]  Elia Gatti,et al.  Pseudo-Haptic Weight: Changing the Perceived Weight of Virtual Objects By Manipulating Control-Display Ratio , 2019, CHI.

[84]  M. Turcsányi-Szabó,et al.  Augmented Reality in Education , 2018, Handbook of Mobile Teaching and Learning.

[85]  Eliot Winer,et al.  Augmented reality via expert demonstration authoring (AREDA) , 2019, Comput. Ind..

[86]  Gordon Wetzstein,et al.  Michelson Holography: Dual-SLM Holography with Camera-in-the-loop Optimization , 2020, ArXiv.

[87]  Ellen Yi-Luen Do,et al.  Ambiotherm: Enhancing Sense of Presence in Virtual Reality by Simulating Real-World Environmental Conditions , 2017, CHI.

[88]  Agata Rozga,et al.  Using electrodermal activity to recognize ease of engagement in children during social interactions , 2014, UbiComp.

[89]  He Zhang,et al.  Digital Twin in Industry: State-of-the-Art , 2019, IEEE Transactions on Industrial Informatics.

[90]  Takuji Narumi,et al.  Meta Cookie+: An Illusion-Based Gustatory Display , 2011, HCI.

[91]  Kenan Bektas Toward A Pervasive Gaze-Contingent Assistance System: Attention and Context-Awareness in Augmented Reality , 2020, ETRA Adjunct.

[92]  Kanchana Thilakarathna,et al.  Security and Privacy Approaches in Mixed Reality , 2018, ACM Comput. Surv..

[93]  Anselmo Lastra,et al.  From Motion to Photons in 80 Microseconds: Towards Minimal Latency for Virtual and Augmented Reality , 2016, IEEE Transactions on Visualization and Computer Graphics.

[94]  Alex Olwal,et al.  Grabity: A Wearable Haptic Interface for Simulating Weight and Grasping in Virtual Reality , 2017, UIST.

[95]  Stefanie Müller,et al.  SensiCut: Material-Aware Laser Cutting Using Speckle Sensing and Deep Learning , 2021, UIST.

[96]  Takuji Narumi,et al.  Pseudo-gustatory display system based on cross-modal integration of vision, olfaction and gustation , 2011, 2011 IEEE Virtual Reality Conference.

[97]  Hiroshi Ishida,et al.  Smelling Screen: Development and Evaluation of an Olfactory Display System for Presenting a Virtual Odor Source , 2013, IEEE Transactions on Visualization and Computer Graphics.

[98]  Gordon Wetzstein,et al.  Learned rotationally symmetric diffractive achromat for full-spectrum computational imaging , 2020 .

[99]  Joohwan Kim,et al.  Foveated AR , 2019, ACM Trans. Graph..

[100]  Ibrahim M. Baggili,et al.  Forensic Analysis of Immersive Virtual Reality Social Applications: A Primary Account , 2018, 2018 IEEE Security and Privacy Workshops (SPW).

[101]  Manjunatha Mahadevappa,et al.  A Novel Technique to Develop Cognitive Models for Ambiguous Image Identification Using Eye Tracker , 2020, IEEE Transactions on Affective Computing.

[102]  Jeremy Bailenson,et al.  Protecting Nonverbal Data Tracked in Virtual Reality , 2018, JAMA pediatrics.

[103]  Mark Roman Miller,et al.  Personal identifiability of user tracking data during observation of 360-degree VR video , 2020, Scientific Reports.

[104]  Eyal Ofek,et al.  Mise-Unseen: Using Eye Tracking to Hide Virtual Reality Scene Changes in Plain Sight , 2019, UIST.

[105]  John C. Tang,et al.  “I just went into it assuming that I wouldn't be able to have the full experience”: Understanding the Accessibility of Virtual Reality for People with Limited Mobility , 2020, ASSETS.

[106]  Andreas Bulling,et al.  Pervasive Attentive User Interfaces , 2016, Computer.

[107]  Evan Suma Rosenberg,et al.  Evaluating immersive experiences during Covid-19 and beyond , 2020, Interactions.

[108]  Krzysztof Krejtz,et al.  Using Microsaccades to Estimate Task Difficulty During Visual Search of Layered Surfaces , 2020, IEEE Transactions on Visualization and Computer Graphics.

[109]  Ben Shneiderman,et al.  Human-Centered Artificial Intelligence: Three Fresh Ideas , 2020, AIS Transactions on Human-Computer Interaction.

[110]  Takuji Narumi,et al.  DeepTaste: Augmented Reality Gustatory Manipulation with GAN-Based Real-Time Food-to-Food Translation , 2019, 2019 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[111]  W. Sato,et al.  Enhanced facial EMG activity in response to dynamic facial expressions. , 2008, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[112]  Hakan Urey,et al.  State of the Art in Stereoscopic and Autostereoscopic Displays , 2011, Proceedings of the IEEE.

[113]  Diane J. Cook,et al.  Smart secure homes: a survey of smart home technologies that sense, assess, and respond to security threats , 2017, Journal of Reliable Intelligent Environments.

[114]  Steve Marschner,et al.  On-The-Fly Print: Incremental Printing While Modelling , 2016, CHI.

[115]  Otmar Hilliges,et al.  Learning Cooperative Personalized Policies from Gaze Data , 2019, UIST.

[116]  Nathaniel I. Durlach,et al.  Telepresence , 1992, Presence: Teleoperators & Virtual Environments.

[117]  Grigore C. Burdea,et al.  A virtual-reality-based telerehabilitation system with force feedback , 2000, IEEE Transactions on Information Technology in Biomedicine.

[118]  Yifan Peng,et al.  Wirtinger holography for near-eye displays , 2019, ACM Trans. Graph..

[119]  Stefan Saroiu,et al.  Home automation in the wild: challenges and opportunities , 2011, CHI.

[120]  Arzu Çöltekin,et al.  GeoGCD: improved visual search via gaze-contingent display , 2019, ETRA.

[121]  Alexandru Dancu,et al.  The Ultimate Display , 2014 .

[122]  Laura Waller,et al.  High resolution étendue expansion for holographic displays , 2020, ACM Trans. Graph..

[123]  David R. Karger,et al.  Squadbox: A Tool to Combat Email Harassment Using Friendsourced Moderation , 2018, CHI.

[124]  U. Huws,et al.  Telework: Towards the Elusive Office , 1990 .

[125]  Sophie Leroy Why is it so hard to do my work? The challenge of attention residue when switching between work tasks , 2009 .

[126]  Rachel A. Albert,et al.  Foveated AR: Dynamically-Foveated Augmented Reality Display , 2019 .

[127]  Qiang Liu,et al.  Ultra-Low Power Gaze Tracking for Virtual Reality , 2017, SenSys.

[128]  Fumio Kishino,et al.  Augmented reality: a class of displays on the reality-virtuality continuum , 1995, Other Conferences.

[129]  Edward H. Glaessgen,et al.  The Digital Twin Paradigm for Future NASA and U.S. Air Force Vehicles , 2012 .

[130]  Patrick Baudisch,et al.  RoMA: Interactive Fabrication with Augmented Reality and a Robotic 3D Printer , 2018, CHI.

[131]  Jon E. Froehlich,et al.  What Do We Mean by “Accessibility Research”?: A Literature Survey of Accessibility Papers in CHI and ASSETS from 1994 to 2019 , 2021, CHI.

[132]  Nezih Mrad,et al.  The role of data fusion in predictive maintenance using digital twin , 2018 .

[133]  Patrick Brézillon,et al.  Facilitating resource sharing and selection in ubiquitous multi-user environments , 2018, Inf. Syst. Frontiers.

[134]  Franz Zotter,et al.  Ambisonics: A Practical 3D Audio Theory for Recording, Studio Production, Sound Reinforcement, and Virtual Reality , 2019 .

[135]  Mark Billinghurst,et al.  An occlusion capable optical see-through head mount display for supporting co-located collaboration , 2003, The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings..

[136]  Frank Fischer,et al.  Simulation-Based Learning in Higher Education: A Meta-Analysis , 2020, Review of Educational Research.

[137]  Marc Alexa,et al.  Directional screens , 2017, SCF.

[138]  G. Gary Wang,et al.  Definition and Review of Virtual Prototyping , 2002, J. Comput. Inf. Sci. Eng..

[139]  John Sweller,et al.  Measuring cognitive load , 2018, Perspectives on Medical Education.

[140]  Pattie Maes,et al.  VMotion: Designing a Seamless Walking Experience in VR , 2018, Conference on Designing Interactive Systems.

[141]  Yifan Peng,et al.  Learned large field-of-view imaging with thin-plate optics , 2019, ACM Trans. Graph..

[142]  Austin Henderson,et al.  Your place or mine? Learning from long-term use of Audio-Video communication , 1996, Computer Supported Cooperative Work (CSCW).