Human Motion Capture

Martin Côté
M.A.Sc. student

Silvain Bériault
M.A.Sc. student

Ashwani Panwar
International internship

Dr. Pierre Payeur
SITE, University of Ottawa


Dr. Gilles Comeau
Dept. of Music, University of Ottawa

Piano Pedagogy Research Laboratory

University of Ottawa

Natural Sciences and Engineering Research Council of Canada

University of Ottawa - Interdisciplinary Initiatives Development Fund

Yamaha Canada Music

This project develops new computer vision technologies for monitoring the complex movements executed by piano players. Piano teachers still widely use subjective and incomplete visual observation of the pianist's posture and gestures to help their students improve their play. This does not provide a quantitative method for evaluating and comparing pianists habits and performance and to detect problematic situations. It has been demonstrated that bad habits can lead to severe piano-playing related health problems (PRHP). This reveals a need for technologies that would contribute in the prevention of such injuries and to the improvement of pedagogical methods, without being too complex to operate.

Existing motion capture technologies used in medicine and movie production remain invasive and cumbersome as they involve massive cabling and/or reflective targets to be installed on the body of the performer. Other approaches rely on contrasting backgrounds or on assumptions about the motion and complexity of the scene. These impositions yield an environment that is foreign to a performer, leading him to behave differently than he would in a more comfortable environment. The limitations of such techniques may also obfuscate key performance markers.

In order to alleviate these constraints, advanced computer vision and image processing approaches are investigated as sensing techniques in the context of piano playing. A multi-camera video monitoring network has been designed along with a complete calibration procedure. An innovative image segmentation technique able to adapt to unconstrained environments has also been developed. The system can perform live motion capture of a piano player without using any marker nor imposing a specific dress code to the pianist. The objective of the first phase of research is to achieve markerless motion capture of a performer that can operate without constraints on the surrounding environment. Early experimental results demonstrated the feasibility of the approach, resulting in accurate colored 3D reconstructions of a subject performing some movement.

Click on image to launch video [XviD codec required]

Future phases are going to investigate the recognition and classification of patterns of movement from the quantitative information extracted from the motion capture system. This will open the door to an exploration of biomechanical impacts of the repetition of some gestures and to the prevention of PRHPs. It will also provide the necessary background to support the development of computerized piano teaching resources that will automatically analyze the movement of the performer's body as well as the depression of keys on the keyboard.

Related Publications

  • S. Bériault, M. Côté, P. Payeur, "Volumetric Modeling with Multiple Cameras for Markerless Motion Capture in Complex Scenes", Proceedings of the IEEE International Instrumentation and Measurement Technology Conference (I2MTC 2008), pp. 359-364, Victoria, BC, 13-15 May 2008. [pdf]

  • M. Côté, P. Payeur, G. Comeau, "Video Segmentation for Markerless Motion Capture in Unconstrained Environments", 3rd International Symposium on Visual Computing (ISVC 2007), Lake Tahoe, NV/CA, LNCS, vol. 4842, pp. 791-800, Springer-Verlag Berlin Heidelberg, 26-28 Nov. 2007. [pdf]

  • S. Bériault, P. Payeur, G. Comeau, "Flexible Multi-Camera Network Calibration for Human Gesture Monitoring", Proceedings of the IEEE International Workshop on Robotic and Sensors Environments (ROSE 2007), pp. 125-130, Ottawa, ON, 12-13 Oct. 2007. [pdf]

  • P. Payeur, M. Côté, G. Comeau, "Les Technologies de l'Imagerie au Service de l'Analyse du Mouvement en Pédagogie du Piano", Recherche en Éducation Musicale, no. 24, pp. 61-87, Aug. 2006. [pdf]

  • M. Côté, P. Payeur, G. Comeau, "Comparative Study of Adaptive Image Segmentation Techniques for Gesture Analysis in Unconstrained Environments", Proceedings of the IEEE International Workshop on Imaging Systems and Techniques (IST 2006), pp. 28-33, Minori, Italy, Apr. 2006. [pdf]

© SMART Research Group, 2008