Shoot & copy: phonecam-based information transfer from public displays onto mobile phones

Large public displays have become pervasive in our everyday lives, but up to now, they are mostly information screens without any interaction possibilities. Users tend to forget what they saw relatively fast after leaving such a display. In this paper, we present a new interaction technique for transferring information from a public display onto a personal mobile phone with its built-in camera. Instead of having to rely on their memory, users simply take a picture of the information of interest. Instead of just storing the image, our system then retrieves the actual data represented on the screen, such as a stock quote, news text, or piece of music. The Shoot & Copy technique does not require visual codes that interfere with shown content or reduce screen real estate. Our prototype allows users to capture an arbitrary region of a standard desktop screen, containing icons, which represent pieces of data. The captured image is then analyzed and a reference to the corresponding data is sent back to the mobile phone. Once the user has time to view the information in more detail, our system allows retrieving the actual data from this reference. We present our prototype and the methods it uses for image processing, as well as an evaluation of our interaction technique illustrating its potential use and applications.