Mixed Reality Teleoperation Console for Robotic Surgical Systems
DOI:
https://doi.org/10.17979/ja-cea.2025.46.12211Keywords:
Realidad Mixta, Cirugía robótica, Teleoperación, Cirugía mínimamente invasivaAbstract
This work presents the design and implementation of an immersive teleoperation console for robotic surgical systems based
on mixed reality (MR). The proposed system integrates virtual reality glasses and a haptic controller for operating a robotic
assistant, such as a surgical suction device, during minimally invasive procedures. The MR console overlays relevant augmented
information onto the real endoscopic image, providing contextual support to the assistant. The architecture supports both direct
teleoperation and supervision of semi-autonomous behaviors. The visualization system is implemented in Unity, supporting both
two-dimensional and three-dimensional views, and is integrated with the ROS operating system, enabling communication with
other components of the system. A use case is described for controlling a surgical suction device, which has been automated by
adapting a conventional aspirator to operate via ROS commands. Finally, an experiment with the complete system is presented,
demonstrating the feasibility of the proposed teleoperation console.
References
Barragan, J. A., Chanci, D., Yu, D., Wachs, J. P., 8 2021. SACHETS: Semiautonomous cognitive hybrid emergency teleoperated suction. 2021 30th IEEE International Conference on Robot and Human Interactive Communication, RO-MAN 2021, 1243–1248. DOI: 10.1109/RO-MAN50785.2021.9515517
Blanco, I. R., Rodríguez, E. G., López-Casado, C., Roldán, M. C., 7 2024. Aspirador quirúrgico autónomo para una intervención de cirugía laparoscópica. Jornadas de Automática (45), 45. DOI: 10.17979/JA-CEA.2024.45.10924
Cremades Pérez, M., Espin Álvarez, F., Pardo Aranda, F., Navinés López, J., Vidal Piñeiro, L., Zarate Pinedo, A., Piquera Hinojo, A. M., Sentí Farrarons, S., Cugat Andorra, E., 5 2023. Realidad aumentada en cirugía hepato-bilio-pancre´atica. Una tecnolog´ıa al alcance de la mano. Cirugía Española 101 (5), 312–318. DOI: 10.1016/J.CIRESP.2022.10.022
Da Col, T., Caccianiga, G., Catellani, M., Mariani, A., Ferro, M., Cordima, G., De Momi, E., Ferrigno, G., de Cobelli, O., 11 2021. Automating Endoscope Motion in Robotic Surgery: A Usability Study on da Vinci-Assisted Ex Vivo Neobladder Reconstruction. Frontiers in Robotics and AI 8, 707704. DOI: 10.3389/FROBT.2021.707704/BIBTEX
Hamann, D., Mortensen, W. S., Hamann, C. R., Smith, A., Martino, B., Dameff, C., Tully, J., Kim, J., Torres, A., 12 2014. Experiences in adoption of teledermatology in Mohs micrographic surgery: Using smartglasses for intraoperative consultation and defect triage. Surgical Innovation 21 (6), 653–654. DOI: 10.1177/1553350614552735
McCullough, M. C., Kulber, L., Sammons, P., Santos, P., Kulber, D. A., 2018. Google glass for remote surgical tele-proctoring in low- And middleincome countries: A feasibility study from Mozambique. Plastic and Reconstructive Surgery - Global Open 6 (12). DOI: 10.1097/GOX.0000000000001999
Nguyen, C. C., Wong, S., Thai, M. T., Hoang, T. T., Phan, P. T., Davies, J., Wu, L., Tsai, D., Phan, H.-P., Lovell, N. H., Do, T. N., 4 2023. Advanced User Interfaces for Teleoperated Surgical Robotic Systems. Advanced Sensor Research 2 (4), 2200036. DOI: 10.1002/ADSR.202200036
Rahimy, E., Garg, S. J., 6 2015. Google glass for recording scleral buckling surgery. JAMA Ophthalmology 133 (6), 710–711. DOI: 10.1001/JAMAOPHTHALMOL.2015.0465,
Raymer, E., MacDermott, A., Akinbi, A., 12 2023. Virtual reality forensics: Forensic analysis of Meta Quest 2. Forensic Science International: Digital Investigation 47, 301658. DOI: 10.1016/J.FSIDI.2023.301658
Rivas-Blanco, I., Perez-del Pulgar, C., López-Casado, C., Bauzano, E., Muñoz, V., 2 2019. Transferring Know-How for an Autonomous Camera Robotic Assistant. Electronics 8 (2), 224. DOI: 10.3390/electronics8020224
Siemens/ros-sharp, 2025. siemens/ros-sharp: ROS# is a set of open source software libraries and tools in C# for communicating with ROS from .NET applications, in particular Unity3D. URL: https://github.com/siemens/ros-sharp
SMARTsurg Project, 2025. SMARTsurg - SMart weArable Robotic Teleoperated Surgery. URL: http://www.smartsurg-project.eu/
Thai, M. T., Phan, P. T., Hoang, T. T., Wong, S., Lovell, N. H., Do, T. N., 8 2020. Advanced Intelligent Systems for Surgical Robotics. Advanced Intelligent Systems 2 (8), 1900138. DOI: 10.1002/AISY.201900138
Wang, W., Song, H., Zhang, Z., Du, Z., 4 2019. Master-slave motion alignment for an open surgical console. International Journal of Medical Robotics and Computer Assisted Surgery 15 (2), e1974. DOI: 10.1002/RCS.1974
Zhao, Y., Mei, Z., Mao, J., Zhao, Q., Liu, G., Wu, D., 1 2022. Remote Vascular Interventional Surgery Robotics: A Literature Review. DOI: 10.36227/TECHRXIV.14994123
Downloads
Published
Issue
Section
License
Copyright (c) 2025 Eva Góngora Rodríguez, Nerea Casado Sánchez, Carmen López-Casado, Irene Rivas Blanco
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
