Digital beethoven — An android based virtual piano

Smart phones with high resolution cameras have enabled innovative computer vision applications that apply techniques like real-time image processing and virtual reality to provide a close approximation of reality. In this paper, we present an Android application that allows users to play a virtual piano by using a keyboard drawn on a piece of paper. The application allows the user to point the camera of a hand held device towards the keyboard, and process image of the paper keyboard in real time. The application then detects fingers placed on a key and after key detection plays the corresponding sounds. Initial results demonstrate the potential of such an application in providing a viable replacement for heavy and expensive instruments.

[1]  Mani Golparvar-Fard,et al.  High-precision vision-based mobile augmented reality system for context-aware architectural, engineering, construction and facility management (AEC/FM) applications , 2013 .

[2]  Rabia Jafri,et al.  Computer vision-based object recognition for the visually impaired in an indoors environment: a survey , 2013, The Visual Computer.

[3]  Zhihan Lv,et al.  Touch-less interactive augmented reality game on vision-based wearable device , 2015, Personal and Ubiquitous Computing.

[4]  Allan Hanbury,et al.  Color based skin classification , 2012, Pattern Recognit. Lett..

[5]  Gwo-Jen Hwang,et al.  An Augmented Reality-based Mobile Learning System to Improve Students' Learning Achievements and Motivations in Natural Science Inquiry Activities , 2014, J. Educ. Technol. Soc..

[6]  Aristotelis Hadjakos,et al.  Pianist Motion Capture with the Kinect Depth Camera , 2012 .

[7]  A. Conci,et al.  Skin Detection using HSV color space , 2009 .

[8]  Georg Essl,et al.  Augmented Piano Performance using a Depth Camera , 2012, NIME.

[9]  Sushmita G. Kamble,et al.  Facial expression based music player , 2016, 2016 International Conference on Advances in Computing, Communications and Informatics (ICACCI).

[10]  Linmi Tao,et al.  Infrared vs. Ultrasonic Finger Detection on a Virtual Piano Keyboard , 2014, ICMC.

[11]  Vladimir Vezhnevets,et al.  A Survey on Pixel-Based Skin Color Detection Techniques , 2003 .

[12]  Jeremy R. Cooperstock,et al.  User-Driven Techniques for the Design and Evaluation of New Musical Interfaces , 2016, Computer Music Journal.

[13]  José M. Chaves-González,et al.  Detecting skin in face recognition systems: A colour spaces study , 2010, Digit. Signal Process..

[14]  I. Zaqout,et al.  Augmented Piano Reality , 2015 .

[15]  Anupam Agrawal,et al.  Vision based hand gesture recognition for human computer interaction: a survey , 2012, Artificial Intelligence Review.

[16]  Dimitrios Buhalis,et al.  Smartphone Augmented Reality Applications for Tourism , 2012 .

[17]  Guojun Lu,et al.  Review of shape representation and description techniques , 2004, Pattern Recognit..

[18]  Anna L. Cox,et al.  Research Methods for Human-Computer Interaction , 2008 .

[19]  Junsong Yuan,et al.  Barehanded music: real-time hand interaction for virtual piano , 2016, I3D.

[20]  Nikolaos G. Bourbakis,et al.  A survey of skin-color modeling and detection methods , 2007, Pattern Recognit..