A framework for engineering gesture-based user interfaces using the Leap Motion controller : development and applications

The continuous improvement of gesture-sensing technologies by sensors, wearable devices, and computer vision is fostering a shift towards more natural user interfaces taking benefit of these technologies. After mastering 2D gesture recognition on contact-based interactive surfaces, researchers and practitioners are turning their focus of interest to contact-less gestures performed in mid-air interaction spaces. The Leap Motion Controller has become a ubiquitous device for capturing contact-less mid-air gestures that led to numerous projects and applications. Since the Leap Motion comes with an Application Programming Interface, the development of gesture-based user interfaces is primarily achieved by hand programming in an opportunistic way, thus impeding maintainability, portability, and compatibility. To satisfy these software quality properties, we conduct a target literature review in Leap Motion gesture recognition to develop a framework for engineering gesture-based user interfaces captured by a Leap Motion that preserves these properties. Its generic and flexible software architecture accommodates several configurations for gesture recognition, testing, and benchmarking of gesture recognizers, as well as the deployment of interactive applications with Leap Motion gesture-based user interfaces. We illustrate the development process on DICOM Viewer, an application for browsing medical images by gestures. Then, we perform a benchmarking of implemented gesture recognizers. Then, we deploy a large-size, real-world application in the context of the Large User Interface (LUI), a project launched by MIT Media Lab. Finally, we present a plan for evaluating the framework and an application deployed based on it and we discuss the benefits and shortcomings of this work before suggesting some future avenues.