Dexterous Robot Manipulation of Deformable Objects with Visual and Tactile Feedback

Félix Nadon
Ph.D. student, 2020-...
M.A.Sc. student, 2018-2019

Stephen Rowlands
M.A.Sc. student

Narmadha Sambandam
M.A.Sc. student

Angel J. Valencia
M.A.Sc. student

Fei Hui
M.A.Sc. student

Jonathan Guillotte-Blouin
UROP internship

Fouad Khalil
Ph.D. student

Nicolas Robin
Summer internship

Dr. Pierre Payeur
EECS, University of Ottawa


Dr. Ana-Maria Cretu
Université du Québec en Outaouais

Dr. Emil Petriu
EECS, University of Ottawa

Ontario Centres of Excellence

Neptec Design Group

Canadian Foundation for Innovation

Ontario Innovation Trust

Modern robotic systems used in industrial, security and space applications require a new generation of autonomous robot manipulators able to perform intelligently on sophisticated manipulation tasks in environments which may be unknown, variable or unstructured. Much research efforts have been devoted to the development of robots that can sense and react to their environment and to the type of object to be handled. Human experience with objects manipulation provides evidence of the powerful capabilities that can be made available by the combination of vision and touch. However, autonomous manipulators still need to be developed to produce similar coordination between these sensory systems to work efficiently in unknown environments and to adapt to unpredicted modifications.

Until now, the vast majority of objects to be manipulated by robots have been assumed to be rigid. The problem of grasping and manipulating 3D deformable objects with a robotic arm and end effector has received much less attention. But in reality, numerous objects are of the latter type, that is, compliant and with alterable shapes. Handling such objects represents a challenging area of research because of the complexity of the interactions that happen between the robot and the object surface which undergoes deformations in response to forces applied by the manipulator.

Applications for the manipulation of non-rigid objects range from advanced manufacturing, food processing, agriculture and even medicine. Simultaneously ensuring the stability of a grasp and the integrity of an object, while undergoing robotic manipulation, represents a major challenge, that must capitalize on multiple sensory inputs, optimize contact points distribution and forces application, all within the constraints imposed by the mechanical assembly of a robotic hand and while operating in a given, often challenging, environment. This research investigates fundamental engineering and computer science solutions to achieve safe, efficient and precise robotic manipulation strategies, encompassing data acquisition, modeling, material characterization, deformation prediction, manipulation optimization, and robotic devices control.

Related Publications

  • A.J. Valencia, F. Nadon, P. Payeur, "Toward Real-Time 3D Shape Tracking of Deformable Objects for Robotic Manipulation and Shape Control", Proceedings of the IEEE SENSORS 2019 Conference, Montreal, QC, 27-30 Oct. 2019. [pdf]

  • F. Nadon, P. Payeur, "Automatic Selection of Grasping Points for Shape Control on Non-Rigid Objects", Proceedings of the IEEE International Symposium on Robotic and Sensors Environments (ROSE 2019), pp. 170-176, Ottawa, ON, 17-18 Jun. 2019. [pdf]

  • F. Nadon, A.J. Valencia, P. Payeur, "Multi-Modal Sensing and Robotic Manipulation of Non-Rigid Objects - a Survey", Robotics, 7(4), 74, 20 Nov. 2018, doi: 10.3390/robotics7040074. [pdf]

  • A.-M. Cretu, P. Payeur, "Harnessing Vision and Touch for Compliant Robotic Interaction with Soft and Rigid Objects", Advanced Interfacing Techniques for Sensors, B. George, J. Kumar Roy, V.J. Kunar, S.C. Mukhopahdhyay (eds), Lecture Series on Smart Sensors, Measurement and Instrumentation, vol. 25, pp. 269-290, Springer International Publishing AG, Switzerland, 5 Apr. 2017. [pdf]

  • F. Hui, P. Payeur, A.-M. Cretu, "Visual Tracking of Deformation and Classification of Non-Rigid Objects with Robot Hand Probing", Robotics, vol. 6, issue 1, art. 5, 17 Mar. 2017. [pdf]

  • F. Hui, P. Payeur, A.-M. Cretu, "In-Hand Object Material Characterization with Fast Level Set in Log-Polar Domain and Dynamic Time Warping", Proceedings of the IEEE International Instrumentation and Measurement Technology Conference (I2MTC 2017), pp. 730-735, Torino, Italy, 22-25 May 2017. [pdf]

  • A.-M. Cretu, P. Payeur, E.M. Petriu, "Soft Object Deformation Monitoring and Learning for Model-Based Robotic Hand Manipulation", IEEE Transactions on Systems, Man and Cybernetics, vol. 42, no 3, pp. 740-753, June 2012. [pdf]

  • F. F. Khalil, P. Payeur, "Dexterous Robotic Manipulation of Deformable Objects with Multi-Sensory Feedback - a Review", Robot Manipulators, Trends and Development, In-Tech (Eds), Chap. 28, 33 pages, ISBN 978-953-307-073-5, March 2010. [pdf]

  • A.-M. Cretu, P. Payeur, E.M. Petriu, "Learning and Prediction of Soft Object Deformation using Visual Analysis of Robot Interactions", Proceedings of the 6th International Symposium on Visual Computing (ISVC 2010), Las Vegas, NV, G. Bebis et al. (Eds), LNCS, vol. 6454, pp. 232-241, Springer-Verlag Berlin Heidelberg, 29 Nov.-1 Dec. 2010. [pdf]

  • F.F. Khalil, P. Payeur, A.-M. Cretu, "Integrated Multisensory Robotic Hand System for Deformable Objects Manipulation", Proceedings of the International Conference on Robotics and Application (RA 2010), pp. 159-166, Cambridge, MA, 1-3 Nov. 2010. [pdf]

  • A.-M. Cretu, E.M. Petriu, P. Payeur, F.F. Khalil, "Estimation of Deformation Object Properties from Shape and Force Measurements for Virtualized Reality Applications", Proceedings of the IEEE International Symposium on Haptic, Audio-Visual Environments and Games (HAVE 2010), pp. 168-173, Tempe, AZ, 16-17 Oct. 2010. [pdf]

  • F.F. Khalil, P. Curtis, P. Payeur, "Visual Monitoring of Surface Deformations on Objects Manipulated with a Robotic Hand", Proceedings of the IEEE International Workshop on Robotic and Sensors Environments (ROSE 2010), pp. 128-133, Tempe, AZ, 15-16 Oct. 2010. [pdf]

  • A.-M. Cretu, E.M. Petriu, P. Payeur, F.F. Khalil, "Deformable Object Segmentation and Contour Tracking in Image Sequences Using Unsupervised Networks", Proceedings of the 7th Canadian Conference on Computer and Robot Vision (CRV 2010), pp. 277-284, Ottawa, ON, 31 May-2 June 2010. [pdf]

  • E.M. Petriu, P. Payeur, A.-M. Cretu, C. Pasca, "Complementary Tactile Sensor and Human Interface for Robotic Telemanipulation", Proceedings of the IEEE International Workshop on Haptic Audio-Visual Environments and Games (HAVE 2009), pp. 164-169, Lecco, Italy, 7-8 Nov. 2009. [pdf]

  • F. Khalil, P. Payeur, "Robotic Interaction with Deformable Objects under Vision and Tactile Guidance - a Review", Proceedings of the IEEE International Workshop on Robotic and Sensors Environments (ROSE 2007), pp. 40-45, Ottawa, ON, 12-13 Oct. 2007. [pdf]

© SMART Research Group, 2020