Challenges in Embedded Vision for Augmented Reality

Augmented Reality (AR) applications hold great promise for mobile users in the near future, but mobile devices cannot yet deliver on this promise. Even the quite substantial processing capabilities of modern mobile devices are not at the level needed for running the latest object recognition, tracking, or rendering methods. Furthermore, the resultant power consumption drains the battery fast. To ensure a great user experience, AR algorithms have to cope with real-world conditions like illumination, jitter, scale, rotation, and noise. The fusion of different optimized AR technologies like 2D or 3D feature tracking, edge detection, gravity awareness, or SLAM should be able to handle these issues to the satisfaction of the end-user but current mobile devices cannot handle these complex algorithms running in real-time and in an “always on–always Augmented mode”. With the “always on–always Augmented” scenario, AR applications need to work on various devices such as smartphones, tablets, PC, etc., and at different form factors such as wearables and head mounted displays. Additionally, technology enablers such as dedicated hardware to accelerate feature tracking and matching that keep the power consumption to an acceptable level, along with easy-to-use software tools and user interfaces designed for non-experts would be the key to provide a low-cost and high volume AR solution. The “always on–always Augmented” scenario also needs considerable cloud support to offload computations and provide a seamless AR experience. To implement these technologies, hardware (HW) architectures have to evolve in parallel to provide efficient resources that can keep power consumption at an acceptable level. In this chapter we discuss the challenges and solutions in embedded processing for a seamless AR user experience in the context of the “always on–always Augmented” use case.