Augmented reality (AR) navigation using a position sensor in endoscopic surgeries relies on the quality of patient–image registration and hand–eye calibration. Conventional methods collect the necessary data to compute separately those two output transformation matrices. However, the AR display setting during the surgery generally differs from that before the surgery. Although conventional methods can identify optimal solutions under initial conditions before the surgery, AR display errors are unavoidable during the surgery. The errors result from the inherent computational complexity of AR processes including the errors accumulation over successive matrix multiplications, and tracking errors of the position sensor.
We proposed a simultaneous optimization of patient–image registration and hand–eye calibration in an AR environment before the surgery. The relationship between the endoscope and a virtual object to overlay is first calculated using an endoscopic image, which also functions as a reference during the optimization. After including the tracking information from the position sensor, patient–image registration and hand–eye calibration are optimized in terms of least-square error. Experiments with synthetic data verified that the proposed method is less sensitive to the computation and the tracking errors. A phantom experiment with a position sensor was also conducted. The accuracy of the proposed method was significantly higher than that of the conventional method. This study demonstrates that the proposed method exhibits substantial potential for improving AR navigation accuracy in the endoscopic surgery.