Immersive AR-Based Wrist Rehabilitation with Visual Robotic Removal via Image Inpainting
DOI:
https://doi.org/10.17979/ja-cea.2025.46.12152Keywords:
Percepción y detección, Procesamiento y sistemas de imágenes biomédicas y médicas, Robótica, Tecnología asistencial e ingeniería de rehabilitación, Trabajo en entornos reales y virtualesAbstract
This work presents an innovative system for wrist rehabilitation that combines an assistive robotic device with an immersive augmented reality (AR) platform. To enhance the patient's experience and support neuroplasticity mechanisms, a visual robot removal algorithm is proposed using image segmentation and inpainting techniques. The system integrates a UNet-based segmentation model and a specifically trained inpainting model, effectively reconstructing the scene realistically after removing the robotic device. The performance of different inpainting models has been evaluated using both traditional and perceptual metrics, showing significant improvements in the quality of the reconstructed images and the immersive experience of the therapy environment. These results pave the way for more effective and engaging rehabilitation, combining the functional benefits of robotics with the realism of augmented reality.
References
Alonso-Linaje, G., Cisnal, A., Fraile, J. C., Turiel, J. P., 2024. Design and analysis of the m3rob: A robotic platform for wrist and hand rehabilitation. IEEE Access 12, 30472–30481.DOI: 10.1109/ACCESS.2024.3369908
Brizuela, G. S., Rodriguez, D. S., de la Fuente Lopez, E., Fraile, J. C., Turiel, J. P., 2022. Segmentación semántica de elementos mecánicos en entornos aumentados de rehabilitación. In: Jornadas de Rob ótica, Educación y Bioingeniera´ıa.
Chen, T., Zhang, J., Hong, Y., Zhang, Y., Zhang, L., 2024. Assessing image inpainting via re-inpainting self-consistency evaluation. In: Proceedings of the 33rd ACM International Conference on Information and Knowledge Management. CIKM ’24. Association for Computing Machinery, New York, NY, USA, p. 291–300. DOI: 10.1145/3627673.3679693
Cisnal, A., Alonso-Linaje, G., Veganzones, M., amd Juan Carlos Fraile, J. P. T., 2023. M3display: Sistema de realidad aumentada para la rehabilitación de la función motora del miembro superior. In: XLI Congreso Anual de la Sociedad Española de Ingeniería Biomédica. URL: http://hdl.handle.net/10317/13479
Corneanu, C., Gadde, R., Martinez, A. M., January 2024. Latentpaint: Image inpainting in latent space with diffusion models. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). pp. 4334–4343.
Fernández-Sánchez, M., Aza-Hernández, A., Verdugo-Alonso, M., 4 2022. Models of public care for the population with acquired brain injury in Spain: a study of the situation by spanish autonomous communities.
Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y., 6 2014. Generative adversarial networks. Science Robotics 3, 2672–2680. DOI: 10.48550/arXiv.1406.2661
Hao, J., Xie, H., Harp, K., Chen, Z., Siu, K.-C., 2022. Effects of virtual reality intervention on neural plasticity in stroke rehabilitation: A systematic review. Archives of Physical Medicine and Rehabilitation 103 (3), 523–541. DOI: 10.1016/j.apmr.2021.06.024
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S., 6 2017. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in Neural Information Processing Systems 2017- December, 6627–6638. DOI: 10.18034/ajase.v8i1.9
Ho, J., Jain, A., Abbeel, P., 6 2020. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems 2020-December. DOI: https://doi.org/10.48550/arXiv.2006.11239
Horé, A., Ziou, D., 2010. Image quality metrics: PSNR vs. SSIM. In: 2010 20th International Conference on Pattern Recognition. pp. 2366–2369. DOI: 10.1109/ICPR.2010.579
INE, 2020. Encuesta de discapacidad, autonomía personal y situaciones de dependencia. 2020. https://www.ine.es/jaxi/Datos.htm?tpx=51614.
Lugmayr, A., Danelljan, M., Romero, A., Yu, F., Timofte, R., Van Gool, L., June 2022. Repaint: Inpainting using denoising diffusion probabilistic models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 11461–11471.
Masmoudi, M., Zenati, N., Izountar, Y., Benbelkacem, S., Haicheur,W., Guerroudji, M. A., Oulefki, A., Hamitouche, C., 2 2024. Assessing the effectiveness of virtual reality serious games in post-stroke rehabilitation: a novel evaluation method. Multimedia Tools and Applications 83, 36175–36202. DOI: 10.1007/s11042-023-17980-5
Polygerinos, P., Wang, Z., Galloway, K. C., Wood, R. J., Walsh, C. J., 2015. Soft robotic glove for combined assistance and at-home rehabilitation. Robotics and Autonomous Systems 73, 135–143, wearable Robotics. DOI: 10.1016/j.robot.2014.08.014
Rathore, S. S., Hinn, A. R., Cooper, L. S., Tyroler, H. A., Rosamond, W. D., 2002. Characterization of incident stroke signs and symptoms. Stroke 33 (11), 2718–2721. DOI: 10.1161/01.STR.0000035286.87503.31
Rego, P. A., Rocha, R., Faria, B. M., Reis, L. P., Moreira, P. M., 1 2017. A serious games platform for cognitive rehabilitation with preliminary evaluation. Journal of Medical Systems 41, 1–15. DOI: 10.1007/S10916-016-0656-5
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B., Jun. 2022. High-Resolution Image Synthesis with Latent Diffusion Models. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Computer Society, Los Alamitos, CA, USA, pp. 10674–10685. DOI: 10.1109/CVPR52688.2022.01042
Ronneberger, O., Fischer, P., Brox, T., 2015. U-net: Convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells W. M., Frangi, A. F. (Eds.), Medical Image Computing and Computer- Assisted Intervention – MICCAI 2015. Springer International Publishing, Cham, pp. 234–241.
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X., 6 2016. Improved techniques for training gans. Advances in Neural Information Processing Systems, 2234–2242. DOI: https://doi.org/10.48550/arXiv.1606.03498
Suvorov, R., Logacheva, E., Mashikhin, A., Remizova, A., Ashukha, A., Silvestrov, A., Kong, N., Goka, H., Park, K., Lempitsky, V., 2022. Resolution-robust large mask inpainting with fourier convolutions. In: 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). pp. 3172–3182. DOI: 10.1109/WACV51458.2022.00323
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan D., Vanhoucke, V., Rabinovich, A., 9 2014. Going deeper with convolutions. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition 07-12-June-2015, 1–9. DOI: 10.1109/CVPR.2015.7298594
Van Rijsbergen, C. J., 1979. Information retrieval. 2nd. newton, ma. Wang, Z., Bovik, A., Sheikh, H., Simoncelli, E., 2004. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13 (4), 600–612. DOI: 10.1109/TIP.2003.819861
World Health Organization, World Bank, 2011. World Report on Disability 2011. World Health Organization, Geneva, nLM classification: HV 1553. URL: https://www.who.int/publications/i/item/9789241564182
Xiao, Z., Kreis, K., Vahdat, A., 12 2021. Tackling the generative learning trilemma with denoising diffusion gans. ICLR 2022 - 10th International Conference on Learning Representations. DOI: https://doi.org/10.48550/arXiv.2112.07804
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., Wang, O., 1 2018. The unreasonable effectiveness of deep features as a perceptual metric. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 586–595. DOI: 10.1109/CVPR.2018.000
Downloads
Published
Issue
Section
License
Copyright (c) 2025 Diego Benavides, Ana Cisnal, Eusebio de la Fuente, Juan Carlos Fraile, Javier Pérez Turiel
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
