Dexterous Robot Manipulation of Deformable Objects with Visual and Tactile Feedback

Fouad Khalil
Ph.D. student

Nicolas Robin
Summer internship

Dr. Pierre Payeur
SITE, University of Ottawa


Dr. Emil Petriu
SITE, University of Ottawa

Ontario Centres of Excellence

Neptec Design Group

Canadian Foundation for Innovation

Ontario Innovation Trust

Modern robotic systems used in industrial, security and space applications require a new generation of autonomous robot manipulators able to perform intelligently on sophisticated manipulation tasks in environments which may be unknown, variable or unstructured. Much research efforts have been devoted to the development of robots that can sense and react to their environment and to the type of object to be handled. Human experience with objects manipulation provides evidences of the powerful capabilities that can be made available by the combination of vision and touch. However, autonomous manipulators still need to be developed to produce similar coordination between these two sensory systems to work efficiently in unknown environments and adapt to unpredicted modifications.

Until now, the vast majority of objects to be manipulated by robots have been assumed to be rigid. The problem of grasping and manipulating 3D deformable objects with a robotic arm and end effector has received much less attention. But in reality, numerous objects are of the latter type, that is, compliant and with alterable shapes. Handling such objects represents a challenging area of research because of the complexity of the interactions that happen between the robot and the object surface which undergoes deformations in response to forces applied by the manipulator.

This project investigates numerous issues that characterize the generic problem of deformable objects manipulation, including: the probing of deformation during robotic interaction, the merge of visual and tactile measurements, the adaptive modeling of the object behavior, and the required control schemes to properly work with the robot dynamics to ensure stable grasp while avoiding damages to the object. A framework is being developed for an autonomous robotic system to safely and reliably manipulate 3D deformable objects under vision and tactile guidance. The approach builds upon technologies originating from 3D machine vision, tactile probing, force/torque sensing, modern control theory and system identification.

Related Publications

  • F. Khalil, P. Payeur, "Robotic Interaction with Deformable Objects under Vision and Tactile Guidance - a Review", Proceedings of the IEEE International Workshop on Robotic and Sensors Environments (ROSE 2007), pp. 40-45, Ottawa, ON, 12-13 Oct. 2007. [pdf]

  • P. Payeur, C. Pasca, A.-M. Cretu, E.M. Petriu, "Intelligent Haptic Sensor System for Robotic Manipulation", IEEE Transactions on Instrumentation and Measurement, vol. 54, no 4, pp. 1583-1592, August 2005. [pdf]

© SMART Research Group, 2008