Simultaneous Robot-World, Sensor-Tip, and Kinematics Calibration of an Underactuated Robotic Hand with Soft Fingers


Simultaneous Robot-World, Sensor-Tip, and Kinematics Calibration of an Underactuated Robotic Hand with Soft Fingers

Ning Tan, Xiaoyi Gu, and Hongliang Ren Senior Member, IEEE

Abstract—Soft robotics is a research field growing rapidly with primary focuses on the prototype design, development of soft robots and their applications. Due to their highly deformable features, it is difficult to model and control such robots in a very precise compared with conventional rigid structured robots. Hence, the calibration and parameter identification problems of an underactuated robotic hand with soft fingers are important, but have not been investigated intensively. In this paper, we present a comparative study on the calibration of a soft robotic hand. The calibration problem is framed as an AX=YB problem with the partially known matrix A. The identifiability of the parameters is analyzed, and calibration methods based on nonlinear optimization (i.e., L-M method and interior-point method) and evolutionary computation (i.e., differential evolution) are presented. Extensive simulation tests are performed to examine the parameter identification using the three methods in a comparative way. The experiments are conducted on the real soft robotic-hand setup. The fitting, interpolating, and extrapolating errors are presented as well.

Index Terms—Soft robotics, calibration and identification, robotic hand, AX=YB, hand-eye calibration, tendon-driven robot

Shape Morphing Microscale Soft Robotic Actuators


The micro actuator has been studied for its application in micro operations such as manipulating cellular aggregate, the tissues or drug delivery. The traditional actuation methods include thermo-mechanical actuation, electromagnetic actuation, electrostatic actuation and pneumatic actuation. Among these actuation methods, pneumatic actuation has the advantage of not generating heat and current during actuation.
We investigate
1) a streamlined and standardized fabrication procedure to make sub-millimeter scale soft pneumatic actuators (SPA) with customizable bending modalities achieved by shape engineering. Preliminary models are also given to interpret width-based shape engineering for customization and to compare the bending angle and radius of curvature measured from the characterization experiments.
2) a new micro pneumatic actuator consisting of two biocompatible materials is designed, fabricated and tested. The actuator has one bending degree of freedom and the largest bending deformation is about 115°.


  • X. Liang; Y. Sun & H. Ren A Flexible Fabrication Approach towards the Shape Engineering of Microscale Soft Pneumatic Actuators IEEE Robotics and Automation Letters, 2016, 1-6
  • X. Liang; C. Lee & H. Ren Towards a Micro Pneumatic Actuator with Large Bending Deformation for Medical Interventions 7th WACBE World Congress on Bioengineering, 6th to 8th July, 2015, Singapore, Springer International Publishing, 2015, 52, 76-79
  • P. M. Khin; J. H. Low; S. Kukreja; H. Ren & R. Yeow Soft Haptics Using Soft Actuator and Soft Sensor BioRob2016, 6th IEEE RAS/EMBS International Conference on Biomedical Robotics and Biomechatronics, June 26-29, 2016, Singapore, 2016

Motion Planning of Flexible Manipulators by Learning from Human Expert Demonstrations


Motion Planning of Multiple-segment flexible soft, and continuum Flexible Manipulators by Learning from Human Expert Demonstrations

Multiple-segment flexible and soft robotic actuators exhibit compliance but suffer from the difficulty of path planning due to their redundant degrees of freedom, although they are promising in complex tasks such as crossing body cavities to grasp objects. We propose a learning from demonstration method to plan the motion paths of flexible manipulators, by statistics machine-learning algorithms. To encode demonstrated trajectories and estimate suitable paths for the manipulators to reproduce the task, models are built based on Gaussian Mixture Model and Gaussian Mixture Regression respectively. The forward and inverse kinematic models of soft robotic arm are derived for the motion control. A flexible and soft robotic manipulator verifies the learned paths by successfully completing a representative task of navigating through a narrow keyhole.


Demo video at:



  • H. Wang; J. Chen; H. Y. Lau & H. Ren Motion Planning of IPMC Flexible Manipulators by Learning from Human Expert Demonstrations ICRA2016, IEEE International Conference on Robotics and Automation, 2016
  • J. Chen; H. REN & A. Lau Learning Reaching Movement Primitives from Human Demonstrations with Gaussian Mixture Regression and Stabilized Dynamical Systems International Conference on Control Science and Systems Engineering ICCSSE 2016, 2016
  • J. Chen; W. Xu; A. Lau & H. REN Towards Transferring Skills to Flexible Surgical Robots with Programming by Demonstration and Reinforcement Learning The Eighth International Conference on Advanced Computational Intelligence (ICACI2016), 2016
  • J. Chen; W. Xu; H. Ren & H. Y. Lau Automate Adaptive Robot Reaching Movement Based on Learning from Human Demonstrations with Dynamical Systems ROBIO2016, 2016

Magnetically Actuated Guide-wire Steering

Project Goals

Guide wires are commonly used to assist surgeons during vascular surgery. Guide wires are usually the first to be placed, eventually allowing easy exchange of surgical tools to the target site. The surgeon has to manually control and manipulate the guide wire to the target region. This process is complicated by the tortuous nature of the vasculature and is made worse as the surgeon has limited vision (constant need to switch between overlay angiographs) and control over the guide wire tip (controlling the distal tip from the proximal end). For the first time towards minimally invasive Arteriovenous Fistula Angioplasty Procedures, we aim to improve the controllability of the in vivo guide wire via the attachment of a magnetic tip to the distal end of the guide wire, which under the control of external magnetic field generators can deflect the magnetic tip. This controlled deflection translates to the selection of entrance angle for the guide wire tip, affording distal control.
This is especially useful for navigating around stenosis which is common for patients with Arteriovenous(AV) Fistula. The repeated use of the fistula during dialysis has been claimed as a cause for the high occurrence of stenosis (due to tissue scaring from access). Fistulas and their grafts are however still preferred over other renal access in most situation due to its lower risk of infections. Our proposed system here can improve the controllability, safety and speed of current procedures and can enable AV-Fistula Angioplasty Procedures to be shifted to out-patient clinics.


People Involved

Student: Bok Seng Yeow
Research fellow: Jinji Sun
Project Investigators: Hongliang Ren, Jackie Ho

Related Publications

Yeow, B. S.; Sun, J.; Ho, J. & Ren, H. Towards Magnetically Actuated Guide-wire Steering in Arteriovenous Fistula Angioplasty Procedures IEEE Conference on Real-time Computing and Robotics (RCAR), IEEE, 2016, best student paper.

Soft Robotic Manipulators: fabrication & applications


Flexible robotic manipulators have been widely used in minimally invasive surgery (MIS) and many other applications requiring closer inspection and operation. Although a variety of manipulators enabled by different mechanism have been developed, few of them can preserve softness, thinness and decent bending capability simultaneously. We develop miniature soft robotic manipulators made of hyper-elastic silicone rubber. Along with the manipulator design, novel fabrication methods are proposed and elaborated. Detailed characterizations are specified to show the bending capability of the manipulator given different air pressure. Specifically, our manipulator, as thin as 6 mm, is able to achieve 360° directional bending, and, when given pressure over 70kPa, it can reach 180° bending angle and around 5mm bending radius easily. Due to its innate compliance and small dimension, this type of robotic manipulator can deliver safe and comfortable interactions with the subjects. More significantly, the novel fabrications in this paper diversify the fabrication methods for soft pneumatic robots and actuators (SPRA) and further scale down their sizes.
Continue reading

Multi-channel iTubot-BAT


The overall objective of this project is to develop and evaluate the key technologies of Transnasal Artificial Tentacles, a new approach involving research on the tentacle-like robotic and sensing technologies. This is aiming to create a compliant, flexible, multiple-sensorized, immersive, intraoperative targeting and guidance system for constrained transnasal endoscopic surgeries in a confined workspace.
Minimally invasive surgery-based nasopharyngeal cancer treatment is promising, but currently, it is not a common treatment choice because of the absence of suitable tools. In this project, a multi-channel concentric tube robot is proposed for the treatment of nasopharyngeal cancer based on natural orifice translumenal endoscopic surgery. The proposed system has three channels, i.e. two manipulation channels and one vision channel, and all the three channels are confined by a 10 mm active sheath. The robot is controlled by human-in-the-loop bimanual teleoperation under active endoscopic guidance. The reduced sheath diameter and the steerable vision channel improve the functionality of the system and distinguish our design from the prior art. The feasibility of the system has been evaluated through a series of simulations and experiments. Results show that the proposed system is capable to conduct cooperative tasks in a confined space and the miniaturized manipulator is suitable for transnasal procedures. Besides, comparisons with other types of flexible surgical robots are discussed to further demonstrate the superiority of the proposed system in the target clinical applications.


Singapore Millennium Foundation
PI: Jan/11/2015 – Jan/10/2018


Yu H, Wu L, Wu K, Lim CM and Ren H (2016), “Development of a Multi-Channel Concentric Tube Robotic System With Active Vision for Transnasal Nasopharyngeal Carcinoma Procedures”, IEEE Robotics and Automation Letters., Jan, 2016. Vol. 1(2), pp. 1172-1178.

A compact continuum tubular robotic system for transnasal procedures


[kad_youtube url=”” ]

Project Goals

Nasopharynx cancer, or nasopharyngeal carcinoma (NPC), is a tumor that originates in the nasopharynx, the uppermost region of the pharynx where the nasal passage and the throat join. It is a common disease occurring to ethnic Chinese people living in or emigrating from southern China; it is also the eighth most frequently occurred cancer among Singaporean men. Traditional posterior nasopharyngeal biopsy using a flexible nasal endoscope has the risks of abrasion and injury to the nasal mucosa and thus causing trauma to the patient. Therefore, the goal of this project is to develop a compact continuum tubular robotic system to achieve collision free nasopharyngeal biopsy.


Fig.1  Illustration of the proposed CTR for nasopharyngeal biopsy.


We developed a compact CTR which is 35 cm in total length, 10 cm in diameter, 2.15 kg in weight, and easy to be integrated with a robotic arm to perform more complicated operations.


Fig.2 The proposed continuum tubular robot


Fig.3 Compact and light weight CTR integrated with a positioning arm for
better conducting surgery

We also developed a 3D printed biopsy needle to equip our robot for transnasal biopsy procedure.

Fig.4  3D printed biopsy needle for transnasal biopsy

The workspace of the robot was analyzed to determine optimized tube parameters.


Fig.5 Workspace comparison for 3-DOF CTR with three initial configurations.
Top: all the outstretched part of the inner tube exposes; Middle: the outstretched part of the inner tube is partially covered by the outer tube; Bottom: the outstretched part of the inner tube is totally covered by the outer tube.

Further more, by using an electromagnetic tracking system, we are able to build a navigation system with shape reconstruction for the tubes.


Fig.6  Shape reconstruction using 3-order Bézier curve fitting


Fig.7 Sensing by EM tracker


Fig.8 Navigation interface


Three groups of experiments were carried out. The first group is to tele-operate the robot to follow a linear path and a circular path. We found that the path following accuracy was about 2 mm.


Fig.9 Tele-operating the robot to follow a linear path and a circular path


Fig.10 Accuracy of the robot following the predefined paths

The second group is to validate the shape reconstruction algorithm. The accuracy of the results is about 1 mm.


Fig.11 Reconstruction setup


Fig.12 Reconstruction error

In the last group of experiments, the robot was tested in a biopsy procedure on a cadaver. The feasibility of the proposed robotic system was validated.


Fig.13  Cadaver experiment setup


Fig.14 Cadaver experiment process

People Involved

Research Fellow: Liao Wu
PhD Student: Keyu Wu
FYP Student: Li Ting Lynette Teo
Intern Student: Jan Feiling and Xin Liu
Project Investigator: Hongliang Ren


[1] Liao Wu, Shuang Song, Keyu Wu, Chwee Ming Lim, Hongliang Ren. Development of a compact continuum tubular robotic system for nasopharyngeal biopsy. Medical & Biological Engineering & Computing. 2016.
[2] Keyu Wu, Liao Wu, Hongliang Ren. Motion planning of continuum tubular robots based on features extracted from statistical atlas. In: Proceedings of 2015 IEEE International Conference on Intelligent Robots and Systems (IROS 2015).
[3] Keyu Wu, Liao Wu, Chwee Ming Lim, Hongliang Ren. Model-free image guidance for intelligent tubular robots with pre-clinical feasibility study: towards minimally invasive trans-orifice surgery. In: Proceedings of 2015 IEEE International Conference on Information and Automation (ICIA 2015). ( best paper finalist)
[4] Benedict Tan, Liao Wu, Hongliang Ren. Prototype development of a handheld tubular curvilinear robot for minimally invasive surgery. In: The 11th Asian Conference on Computer Aided Surgery (ACCAS 2015).
[5] Keyu Wu†, Liao Wu†, Hongliang Ren. An image based targeting method to guide a curvilinear concentric tube robot. In: Proceedings of 2014 IEEE International Conference on Robotics and Biomimetics (ROBIO 2014). Bali, Indonesia, 2014: 386-391 († equally contributed author).


This is a local copy of the website:

Home-Based Self-Administered Nasopharynscopy

Nasopharynscope is a valuable tool in diagnosing Nasopharyngeal carcinoma in patients since 84% of patients display ulcerations.

Aim: To provide a home-based, affordable and easy-to-use diagnosing kit for detecting Nasopharyngeal Carcinoma

Key features of 5th Generation

  • Clear viewing with a specially designed camera lens
  • Secure extension and contraction lock
  • Tight fit between nylon strings to ensure good power transmission 
  • Diamond cuts to enhance bending capabilities
  • Optical zoom of up to 5 mm due to shooting mechanism

Bending Capability: >90 degrees

Extension and contraction to evade obstacles

Overall Demonstration of Bending and Zooming capability


  • Outer Diameter: 7 mm
  • Length of extension portion:25 mm
  • Length of bending segment: 25mm
  • Minimum inserted length: 11-15 cm
  • Gear box: 30 by 20 by 30mm
  • Bending angle: 90 degrees bend per side
  • Distal tip mechanism: Optical zoom
  • Material: Polyurethane (Biocompatible)
  • Stent design: Flexibility
  • Flexible guiding tube

Technical Advantages

  • Large bending angle
  • Extending the camera using the spring mechanism to obtain better optical viewing up to 5mm
  • Endoscope is very flexible with the stent design
  • Able to control the bending of the body segments using cable driven mechanism
  • The bending of the endoscope at the entrance can be controlled flexibly by the guiding shaft


From Left: Mr Teo Jing Chun, Dr Ren Hongliang, Mr Un Weiyang, Miss Soh Yan Bing, Mr Ong Jun Hao Edmund
Foreground: Mr Yeow Bok Seng

ETH Image Based Visual Servoing to Guide Flexible Robots

Video Demo

Eye-To-Hand Image Based Visual Servoing to Guide Flexible Robots

Project goals

Flexible robots including active cannula or cable driven continuum robots are typically suitable for such minimally invasive surgeries because they are able to present various flexible shapes with great dexterity, which strengthens the ability of collision avoidance and enlarges the reachability of operation tools. Using model based control method will lead to artificial singularities and even inverted mapping in many situations because the models are usually developed in free space and cannot perform effectively in constrained environments. Therefore, the goal of this project is control the motion of a tentacle-like curvilinear concentric tube robot by model-less visual servoing.


A two-dimensional planar manipulator is constructed by enabling only the three translation inputs of a six DOF concentric tube robot. As shown in Fig. 1, the concentric tube manipulator is controlled using a PID controller and the images captured by an uncalibrated camera are used as visual feedback.

Fig. 1. The experimental setup includes a concentric tube robot, a camera, a laptop, a marker and a target.

The visual tracking of the concentric tube robot is based on shape detection. The circular marker is attached to the tip of the concentric tube robot and a square target is given for the tip to trace. During the experiments, the coordinates of the marker centroid and target centroid are calculated while the next target position is calculated at the same time as shown in Fig. 2.

Fig. 2. Working mechanism of the system. Top: translations of the three tubes. Bottom: marker, final target and the next target position on the image plane.


Fig. 3. Overview of the control algorithm. The Jacobian matrix is estimated based on the measurements of each incremental movement detected from the camera.

The framework of the controlling the robot is shown in Fig. 3. The initial Jacobian matrix is acquired by running each individual motor separately and measuring the change of tip position of the robot in the image space. Then the optimal control is achieved by solving a typical redundant inverse kinematics. And finally the Jacobian matrix is continuously estimated based on the measured displacements.


To evaluate the proposed model-less algorithm, a simulation was carried out on MATLAB first. The desired and actual trajectory was shown in Fig. 4, from which it could be seen that the robot succeeded in following the reference trajectory and reaching the target position.

Fig. 4. Simulation of using the proposed algorithm to control a concentric tube robot.

The proposed algorithm was also implemented on a physical concentric tube robot in free space. It was found the robot was able to reach goal with zero steady state error in all trials as shown in Fig. 5.

Fig. 5. The concentric tube robot is able to reach a desired goal using the proposed method. Top: the motion of the robot. Bottom: the reference and actual trajectories of two experiments.

People involved

Staff: Keyu WU, Liao WU
PI: Hongliang REN


1. Keyu Wu, Liao Wu and Hongliang Ren, “An Image Based Targeting Method to Guide a Tentacle-like Curvilinear Concentric Tube Robot”, ROBIO 2014, IEEE International Conference on Robotics and Biomimetics, 2014.

3D Ultrasound Tracking and Servoing of Tubular Surgical Robots


[Pediatric Cardiac Bioengineering Lab of Children’s Hospital Boston, Harvard Medical School, USA]
[Philips Research]


Ultrasound imaging is a useful modality for guiding minimally invasive interventions due to its portability and safety. In cardiac surgery, for example, real-time 3D ultrasound imaging is being investigated for guiding repairs of complex defects inside the beating heart. Substantial difficulty can arise, however, when surgical instruments and tissue structures are imaged simultaneously to achieve precise manipulations. This research project includes: (1) the development of echogenic instrument coatings, (2) the design of passive instrument markers, and (3) the development of algorithms for instrument tracking and servoing. For example, a family of passive markers has been developed by which the position and orientation of a surgical instrument can be determined from a single 3D ultrasound volume using simple image processing. Marker-based estimates of instrument pose can be used in augmented reality displays or for image-based servoing.
For example, a family of passive markers has been developed by which the position and orientation of a surgical instrument can be determined from a single 3D ultrasound volume using simple image processing. Marker-based estimates of instrument pose can be used in augmented reality displays or for image-based servoing. The design principles for marker shapes ensure imaging system and measurement uniqueness constraints are met. Error analysis is used to guide marker design and to establish a lower bound on measurement uncertaintanty. Experimental evaluation of marker designs and tracking algorithms demonstrate a tracking accuracy of 0.7 mm in position and 0.075 rad in orientation.
Another example is to investigate the problem of automatic curve pattern detection from 3D ultrasound images, because many surgical instruments are curved along the distal end during operation, such as continuum tube robot, and catheter insertion etc. We propose a two-stage approach to decompose the six parameter constant-curvature curve estimation problem into a two stage parameter estimation problems: 3D spatial plane detection and 2D circular pattern detection. The algorithm includes an image-preprocessing pipeline, including thresholding, denoising, connected component analysis and skeletonization, for automatically extracting the curved robot from ultrasound volumetric images. The proposed method can also be used for spatial circular or arc pattern recognition from other volumetric images such as CT and MRI.
Additional related information at [Pediatric Cardiac Bioengineering Lab of Children’s Hospital Boston, Harvard Medical School]