A multimodal gesture-based virtual interactive piano system using computer vision and a motion controller

Everyday life activities rely on technology and in order to compliment modern day lifestyle, technology has to be either portable or interoperable. In this paper, we propose a close to real-time, gesture-based virtual piano system, using computer vision and motion tracking, with the notion of being accessible in different environments. The solution focuses on capturing a user's hand movements using sensors such as a camera and a motion controller. The system captures the information as the user interacts with the device, in order to simulate the use of an acoustic piano. Furthermore, this paper presents a model, which serves as a blueprint that highlights the important system processes. The model is validated by a prototype in order to assess the usability and viability of the system. The solution has achieved promising performance results in terms of accuracy, related to percentage of false-positives and false-negatives produced. The solution attempts to retain the comfort and efficiency of a traditional acoustic piano, while being accessible in any environment where the device can placed and interacted with.