Surgical Planning

Depth Estimation for Real-time 3D Annotation in Transnasal Surgery

Surgical annotation promotes effective communication between medical personnel during surgical procedures. However, existing approaches to 2D annotations are mostly static with respect to a display. In this work, we propose a method to achieve 3D annotations that anchor rigidly and stably to target structures upon camera movement in a transnasal endoscopic surgery setting. This is accomplished through intra-operative endoscope tracking and monocular depth estimation. A virtual endoscopic environment is utilized to train a supervised depth estimation network. An adversarial network transfers the style from the real endoscopic view to a synthetic-like view for input into the depth estimation network, wherein frame-wise depth can be obtained in real time. Accuracy in terms of estimated depth and stability of the proposed 3D annotation system is evaluated qualitatively and quantitatively.

The training data and test data can be downloaded here.


Advanced pre-operative imaging modalities such as ultrasonography (US), computed tomography (CT) and magnetic resonance imaging (MRI) can provide anatomy and pathology information, which is indispensable and crucial for a detailed diagnosis and treatment plan. 3D anatomical models can be readily visualized for simulation of surgical intervention. Our group is interested in patient-specific surgical planning for computer-assisted surgery. Optimization of cardiac mapping strategy for radiofrequency catheter ablation is one of the examples. We also adopt timely 3D-printing technologies to fabricate patient-specific models using materials that mechanically mimic soft tissues and challenging anatomical features. Such tangible 3D-printed models not only can give better insight for surgical planning, but also allow simulation of actual procedure, as well as validation of virtually optimized surgery plan in pre-clinical settings, eventually enhancing the quality of surgical planning for safer and more efficacious interventions.


[1]  Y. Feng, Z. Guo, Z. Dong, X.Y. Zhou, K.W. Kwok, S. Ernst, S.L. Lee, "An Efficient Cardiac Mapping Strategy for Radiofrequency Catheter Ablation with Active Learning," International Journal of Computer Assisted Radiology and Surgery (IJCARS), vol. 12, no. 7, pp. 1199-1207, 2017.  Detail


[2]  K.C.Y. So, Y. Fan, L. Sze, K.W. Kwok, A.K.Y. Chan, G.S.H. Cheung, A.P.W. Lee, "Using Multi-material 3D Printing for Personalized Planning of Complex Structural Heart Disease Intervention," Journal of the American College of Cardiology (JACC): Cardiovascular Intervention, vol. 10, no. 11, 2017.   Detail

[3]  Y. Fan, K.W. Kwok, Y. Zhang, G. S.H. Cheung, A. K.-Y. Chan, and A.P.W. Lee, "Three-dimensional printing for planning occlusion procedure for a double-lobed left atrial appendage," Circulation: Cardiovascular Interventions, vol. 9, no. 3, p. e003561, 2016.  Detail

[4]  M.C.W. Leong, K.H. Lee, B.P.Y. Kwan, Y.L. Ng, Z. Liu, N. Navab, W. Luk, K.W. Kwok, “Performance-aware Programming for Intraoperative Intensity-based Image Registration on Graphics Processing Units”, International Journal for Computer Assisted Radiology and Surgery (IJCARS), vol. 16, no. 3, 375-386, 2021.   Detail


Intensity-based image registration is a powerful tool to resolve image misalignments. The Diffeomorphic Log-Demons is a well-established, Demons-based registration method. This open-sourced experimental tool is a CUDA-enabled implementation of the algorithm. We employ performance-aware programming techniques to ensure full device utilization. Preliminary results show this tool can speed up the computation by two orders of magnitudes.

The source files of this registration tool can be downloaded here.