Month: December 2013

Surgical Tracking Based on Stereo Vision and Depth Sensing

  By , ,

Project Goals:

The objective of this research is to incorporate multiple sensors at broad spectrum, including stereo infrared (IR) cameras, color (or RGB) cameras and depth sensors to perceive the surgical environment. Features extracted from each modality can contribute to the cognition of complex surgical environment or procedures. Additionally, their combination can provide higher robustness and accuracy beyond what is obtained from single sensing modality. As a preliminary study, we propose a multi-sensor fusion approach for localizing surgical instruments. We developed an integrated dual Kinect tracking system to validate the proposed hierarchical tracking approach.

Approaches:

This project considers the problem of improving the surgical instrument tracking accuracy by multi-sensor fusion technique in computer vision. We proposed a hierarchical fusion algorithm for integrating the tracking results from depth sensor, IR camera pair and RGB camera pair. Fig. 1 summarized the algorithm involved in this project. It can be divided into the “low-level” and the “high-level” fusion.

Fig. 1 Block diagram of hierarchical fusion algorithm.


The low-level fusion is to improve the speed and robustness of marker feature extraction before triangulating the tool tip position in IR and RGB camera pair. The IR and RGB camera are modeled as pin-hole cameras.  The depth data of the tool can be used as a priori for marker detection. The working area of the tracking tool is supposed to be limited in a reasonable volume v(x, y, z) that can be used to refine the search area for feature extraction, which could reduce the computational cost for real-time applications.
The high-level fusion is to reach a highly accurate tracking result by fusing two measurements. We employ the covariance intersection (CI) algorithm to estimate a new tracking result with less covariance.

Experiments

To demonstrate the proposed algorithm, we designed a hybrid marker-based tracking tool (Fig. 2) that incorporates the cross-based feature in visible modality and retro-reflective marker based feature in infra-red modality to get a fused tracking of the customized tool tip. To evaluate the performance of the proposed method, we employ two Kinects to build the experimental setup. Fig. 3 shows the prototype of multi-sensor fusion tracker for the experiment, which indicates that the CI-based fusion approaches obviously tend to be better than the separate IR tracker or RGB tracker.  The mean error and deviation of the fusion algorithm are all improved.
Hybrid marker

Fig. 3 Dual Kinect tracking system

People Involved

Staffs: Wei LIU, Shuang SONG, Andy Lim
Advisor: Dr. Hongliang Ren
Collaborator: Wei ZHANG

Publications

[1] Ren, H.; LIU, W. & LIM, A. Marker-Based Instrument Tracking Using Dual Kinect Sensors for Navigated Surgery IEEE Transactions on Automation Science and Engineering, 2013
[2] Liu, W.; Ren, H.; Zhang, W. & Song, S. Cognitive Tracking of Surgical Instruments Based on Stereo Vision and Depth Sensing, ROBIO 2013, IEEE International Conference on Robotics and Biomimetics, 2013

Related FYP Project

Andy Lim: Marker-Based Surgical Tracking With Multiple Modalities Using Microsoft Kinect

References

[1] H. Ren, D. Rank, M. Merdes, J. Stallkamp, and P. Kazanzides, “Multi-sensor data fusion in an integrated tracking system for endoscopic surgery,” IEEE Transactions on Information Technology in Biomedicine, vol. 16, no. 1, pp. 106 – 111, 2012.
[2] W. Liu, C. Hu, Q. He, and M.-H. Meng, “A three-dimensional visual localization system based on four inexpensive video cameras,” in Information and Automation (ICIA), 2010 IEEE International Conference on. IEEE, 2010, pp. 1065–1070.
[3] F. Faion, S. Friedberger, A. Zea, and U. D. Hanebeck, “Intelligent sensor-scheduling for multi-kinect-tracking,” in Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on. IEEE, 2012, pp. 3993–3999.

FYP: Surgical Tracking With Multiple Microsoft Kinects

  By ,

FYP Project Goals

The aim of this project is to perform tracking of surgical instruments utilizing the Kinect sensors. With the advances in computing and imaging technologies in the recent years, visual limitations during surgery such as those due to poor depth perception and limited field of view, can be overcome by using computer-assisted systems. 3D models of the patient’s anatomy (obtained during pre-operative planning via Computed Tomography scans or Magnetic Resonance Imaging) can be combined with intraoperative information such as the 3D pose and orientation of surgical instruments. Such a computer-assisted system will reduce surgical mistakes and help identify unnecessary or imperfect surgical movements, effectively increasing the success rate of the surgeries.
For computer-assisted systems to work, accurate spatial information of surgical instruments is required. Most surgical tools are capable of 6 degrees of freedom (6DoF) movement, which includes the translation in the x, y, z- axes as well as the rotation about these axes. The introduction of Microsoft Kinect sensor raises the possibility of an alternative optical tracking system for surgical instruments.
This project’s objective would be the development of an optical tracking system for surgical instruments utilising the capabilities of the Kinect sensor. In this part of the project, the focus will be on marker-based tracking using the Kinect sensor.

Approach

  • The setup for the tracking of surgical instruments consists of two Kinects placed side by side with overlapping field of views.
  • The calibration board used to find out the intrinsic camera parameters as well as the relative position of the cameras. This allows us to calculate the fundamental matrix, which is essential for epipolar geometry calculations used in 3D point reconstruction. (a) without external LED illumination (b) with LED illumination. The same board is used for RGB camera calibration.
  • Seeded region growing allows the segmentation of retro-reflective markers from the duller background. The algorithm is implemented through OpenCV.
  • Corner detection algorithm: the cornerSubPix algorithm from OpenCV is used to refine the position of the corners. This results in sub-pixel accuracy of the corner position.

Current Results

  • The RMS error for IRR and checkerboard tracking ranges from 0.37 to 0.68 mm and 0.18 to 0.37 mm respectively over a range of 1.2 m. Checkerboard tracking is found to be more accurate. Error increases with distance from camera.
  • The jitter for the checkerboard tracking system was investigated and it was found to range from 0.071 mm to 0.29 mm over the range of 1.2 m.
  • (dots) Measurement of jitter plotted against the distance from the left camera. (line) the data is fitted to a polynomial of order 2 to analyze how jitter varies with depth.

 

People Involved

FYP Student: Andy Lim Yong Mong
Research Engineer: Liu Wei
Advisor: Dr. Ren Hongliang

Related Project

Surgical Tracking Based on Stereo Vision and Depth Sensing

References

[1] Sun, W., Yang, X., Xiao, S., & Hu, W. (2008). Robust Checkerboard Recognition for Efficient Nonplanar Geometry Registration in Projector-camera Systems. Proceedings of the 5th ACM/IEEE International Workshop on Projector camera systems. ACM.
[2] R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision, 2 ed., Cambridge: Cambridge University Press, 2003.
[3] Q. He, C. Hu, W. Liu, N. Wei, M. Q.-H. Meng, L. Liu and C. Wang, “Simple 3-D Point Reconstruction Methods With Accuracy Prediction for Multiocular System, “IEEE/ASME Transactions on Mechatronics, vol. 18, no. 1, pp. 366-375, 2013

Statistical Humerus Implants and Associated Intramedullary Robotics

  By ,

Project Goals

The sizes of current off-the-shelf humerus implants are unable to accommodate Asian patients since they are mainly produced for American and European populations according to locally collected data. By creating statistical humerus atlases based on Asian data, gender-specific and region-specific humeral implants can be developed by considering the characteristics of the statistical atlas constructed in order to improve stability of the fixation and avoid related complications. Besides, it is envisioned that the statistical atlas can serve as a critical reference for development and evaluation of robots in surgical procedures. Particularly, for the surgical and interventional procedures in the confined and rotated intramedullary space, the curvature and shape statistics of internal humerus canal is of great significance for the dedicatedly design of snake-like curvilinear tubular robot.

In this project, an efficient way has been demonstrated to construct statistical atlas by adopting an efficient alignment algorithm with improved efficiency and good accuracy. The constructed humerus atlas is then regarded as the reference for design of various humerus implants and development of snake-like concentric tube robots.

Approach

Statistical Atlas Construction: A three-step algorithm is adopted in statistical atlas construction, including segmentation, alignment and principal component analysis (PCA). Segmentation is to extract the desired surface mesh information of the humeri from the raw CT data and alignment is to align all the samples. The final step is to perform the principal component analysis of the shapes and represent the statistical model using principal components.

Creation and application of a statistical humerus atlas

Creation and application of a statistical humerus atlas

Centerline Extraction for Intramedullary Robot Design: The Laplacian-Based Contraction Method is adopted to extract the centerline of the humerus atlas. The purpose is to explore the intramedullary structure of the humerus since the curvature and shape statistics of internal humerus canal is significant for the design of snake-like curvilinear tubular robot.

Centerline Extraction Process

Centerline Extraction Process

Curvature Analysis for Design of Humerus Implants: The maximum principal curvature is depicted in the below Figure. The curvature analysis is to study the statistical surface curvature of the humerus atlas, in order to assist the design of humerus implants such as proximal and distal humerus locking plates, both used in orthopaedic trauma fixation.

Principal curvature of the statistical humerus atlas

Principal curvature of the statistical humerus atlas

Current Results

By adopting the novel atlas construction algorithm, the statistical humerus atlas is constructed as shown in the below figure, where the shape variation is along the first three principal components (PCs) and each row is generated by varying the shape with -3 to +3 standard deviations.

The variation along the first principal component is shown here (click to view the animation). From -3std to +3std, the length of the humerus model is increased while the width is decreased.

Shape variation along the 1st principal component

Shape variation along the 1st principal component

Moreover, by analyzing the characteristics of the humerus atlas, the intramedullary continuum robot design and the proximal humerus locking plate are depicted in the following figures.

Design of intramedullary continuum robot based on the statistical humerus atlas

Design of intramedullary continuum robot based on the statistical humerus atlas

Proximal humerus locking plate

Proximal humerus locking plate

People Involved

Staff: Keyu WU
Advisor: Hongliang REN
Clinicians: Keng Lin Wong, Zubin Jimmy Daruwalla, Diarmuid Murphy, National University Hospital

Publications

  • Wu K, Wong FKL, Ng SJK, Quek ST, Zhou B, Murphy D, Daruwalla ZJ and Ren H (2015), “Statistical atlas-based morphological variation analysis of the asian humerus: Towards consistent allometric implant positioning”, International Journal of Computer Assisted Radiology and Surgery. Vol. 10(3), pp. 317-327. Springer Berlin Heidelberg
  • Wu K, Daruwalla ZJ, Wong FKL, Murphyand D and Ren H (2015), “Development and Selection of Asian-specific Humeral Implants based on Statistical Atlas: Towards Planning Minimally Invasive Surgery”, International Journal of Computer Assisted Radiology and Surgery. Vol. 10(8), pp. 1333-1345. Springer Berlin Heidelberg.
  • Wu, K.; Wong, F. K. L.; Daruwalla, Z. J.; Murphy, D. & Ren, H. Statistical Humerus Atlas for Optimal Design of Asian-Specific Humerus Implants and Associated Intramedullary Robotics, ROBIO 2013, IEEE International Conference on Robotics and Biomimetics, 2013; (Best Paper Finalist award)

Supporting materials

Supplementary information for

Statistical Atlas Based Morphologic Variation Analysis of the Asian Humerus: Towards Consistent Allometric Implant Positioning

K. Wu, K. L. Wong, S. J. K. Ng, S. T. Quek, B. Zhou, D. P. Murphy, Z. J. Daruwalla, H. Ren*

 

All data are published in .nii format, which can be opened by ITK-SNAP (http://www.itksnap.org/pmwiki/pmwiki.php) and 3D Slicer (http://www.slicer.org/).

Table 1. Information of the subjects.

Study Number

Male/Female

Age

Race

Weight (kg)

Height (cm)

Exercise habit:

S = sedentary,

A = active

Left/Right:

L = left,

R = right

5260823

F

61

C

67.9

143

S

L

5190438

M

16

C

55.6

160

A

R

5186672

M

63

C

67.5

160

A

L

5172526

M

50

C

61.0

170

S

L

5018737

F

77

C

57.9

151

S

R

4636229

M

54

C

42.8

161

S

L

4585165

M

38

C

60.3

171

A

L & R

4481609

F

83

C

60.1

149

S

L

4448930

F

49

I

73.7

150

S

R

4320030

M

69

C

63.7

163

S

L

4111802

M

63

C

57.4

160

S

L

3969853

F

61

C

64.3

157

A

R

3564131

M

19

C

97.5

167

A

L

3495415

M

74

M

57.1

175

S

L

3827904

M

33

M

64.2

170

A

L

3722692

F

73

I

57.3

151

S

L

3768879

M

20

C

65.0

170

A

L

3771777

F

74

C

53.3

165

S

L & R

3159961

M

51

C

60.0

170

A

L

3273883

F

80

C

54.8

155

S

L

3384844

F

71

C

53.1

155

S

R

3353378

F

78

M

40.0

140

S

L

3271006

F

59

M

60.0

151

S

L

3189139

M

47

C

72.0

179

A

R

3132590

M

22

M

79.0

168

A

L

3143094

M

23

C

60.6

162

A

R

3282457

F

90

C

41.2

150

S

L

3296805

M

57

M

61.2

160

S

L

3326396

M

21

C

76.0

175

A

L

3338591

F

61

C

56.0

153

S

L & R

3060290

M

21

M

73.0

160

A

L

5581085

F

49

M

81.9

159

S

L & R

5691045

F

45

C

45.1

161

A

L

5731852

F

87

C

58.2

150

S

R

5733267

F

57

C

56.4

142

S

L & R

5622431

M

64

I

52.9

162

A

R

6007641

M

44

I

74.2

165

A

L

6013682

F

74

C

52.0

153

S

R

5504982

F

24

C

54.0

153

A

L

image001L
Fig. 1. Statistical male humerus atlas (left). The shape variation is along the first three principal components (PCs) and each row is generated by varying the shape with standard deviations ranging from -3√(λ_k ) to +3√(λ_k ).

image004R
Fig. 2. Statistical female humerus atlas (left). The shape variation is along the first three principal components (PCs) and each row is generated by varying the shape with standard deviations ranging from -3√(λ_k ) to +3√(λ_k ).

References

[1] H. Ren, N. V. Vasilyev, and P. E. Dupont, “Detection of curved robots using 3d ultrasound,” in Intelligent Robots and Systems (IROS), 2011 IEEE/RSJ International Conference on. IEEE, 2011, pp. 2083–2089.
[2] H. Ren and P. E. Dupont, “Artifacts reduction and tubular structure enhancement in 3d ultrasound images,” in International Conference of the IEEE Engineering in Medicine and Biology Society, EMBC, 2011.
[3] G. Chintalapani, L. M. Ellingsen, O. Sadowsky, J. L. Prince, and R. H. Taylor, “Statistical atlases of bone anatomy: construction, iterative improvement and validation,” in Medical Image Computing and Computer-Assisted Intervention–MICCAI 2007. Springer, 2007, pp. 499–506.
[4] H. Ren and P. E. Dupont, “Tubular enhanced geodesic active contours for continuum robot detection using 3d ultrasound,” in IEEE International Conference on Robotics and Automation, ICRA ’12, 2012.
[5] X. Kang, H. Ren, J. Li, and W.-P. Yau, “Statistical atlas based registration and planning for ablating bone tumors in minimally invasive interventions,” in Robotics and Biomimetics (ROBIO), 2012 IEEE International Conference on. IEEE, 2012, pp. 606–611.
[6] J. Cao, A. Tagliasacchi, M. Olson, H. Zhang, Z. Su, “Point Cloud Skeletons via Laplacian Based Contraction,” in Shape Modeling International Conference (SMI), pp. 21-23, 2010.