Autonomous active exploration using ROS2 and Reinforcement Learning
DOI:
https://doi.org/10.17979/ja-cea.2026.47.13735Keywords:
Reinforcement learning control, Information and sensor fusion, Autonomous robotic systems, Autonomous Mobile Robots, Intelligent robotics, Map buildingAbstract
This paper presents an autonomous active exploration and localization system (A-SLAM) based on Deep Reinforcement Learning for an iCreate3 differential robot. For implementation in continuous environments, the TD3 algorithm is used to optimise trajectories that maximise map completeness in unknown environments using continuous parametric control policies. We propose a complete architecture that integrates TD3 with ROS2, the iCreate3 robot, and an RPLIDAR-C1 sensor. This architecture uses a state vector that integrates LiDAR readings, a dual map representation —a local egocentric representation of occupancy and a global exploration representation— and robot position information. The LiDAR and map information is processed using convolutional neural networks to reduce its dimensionality and extract the most relevant spatial features. The system was implemented in ROS2 Humble and validated in the Gazebo Classic simulator. The results obtained after 11,000 training episodes achieved explorations exceeding 95% in map completeness.
References
Alcalde, M., Ferreira, M., González, P., Andrade, F., Tejera, G., 2022. Daslam: Deep active slam based on deep reinforcement learning. In: 2022 Latin American Robotics Symposium (LARS), 2022 Brazilian Symposium on Robotics (SBR), and 2022 Workshop on Robotics in Education (WRE). pp. 282–287. DOI: 10.1109/LARS/SBR/WRE56824.2022.9996006
Arroyo de la Torre, V., Barea Navarro, R., 2021. Aplicación de técnicas de deep reinforcement learning mediante agente cnn-ddpg a la conducción autónoma. Universidad de Alcalá, España, trabajo de fin de grado. URL: https://ebuah.uah.es/dspace/handle/10017/49476
Chen, F., Martin, J. D., Huang, Y.,Wang, J., Englot, B. J., 2020. Autonomous exploration under uncertainty via deep reinforcement learning on graphs. CoRR abs/2007.12640. URL: https://arxiv.org/abs/2007.12640
Cimurs, R., Suh, I. H., Lee, J. H., 2021. Goal-driven autonomous exploration through deep reinforcement learning. URL: https://arxiv.org/abs/2103.07119
Du, S., Duan, T., Yang, X., Gong, X., 2025. Review and frontier exploration of active slam. In: 2025 4th Conference on Fully Actuated System Theory and Applications (FASTA). pp. 2850–2855. DOI: 10.1109/FASTA65681.2025.11138631
Fernandez-Fernandez, R., Victores, J. G., Balaguer, C., 2023. Deep robot sketching: An application of deep q-learning networks for human-like sketching. Cognitive Systems Research 81, 57–63. URL: https://www.sciencedirect.com/science/article/pii/S1389041723000372 DOI: https://doi.org/10.1016/j.cogsys.2023.05.004
Fujimoto, S., van Hoof, H., Meger, D., 2018. Addressing function approximation error in actor-critic methods. CoRR abs/1802.09477. URL: http://arxiv.org/abs/1802.09477
Li, H., Zhang, Q., Zhao, D., 07 2020. Deep reinforcement learning based automatic exploration for navigation in unknown environment. DOI: 10.48550/arXiv.2007.11808
Li, Z., Xin, J., Li, N., 2023. Autonomous exploration and mapping for mobile robots via cumulative curriculum reinforcement learning. URL: https://arxiv.org/abs/2302.13025
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N. M. O., Erez, T., Tassa, Y., Silver, D.,Wierstra, D., 2015. Continuous control with deep reinforcement learning. arXiv: Learning. URL: https://api.semanticscholar.org/CorpusID:16326763
Llobera, M. A., Placed, J. A., Paula, M. D., Cristóforis, P. D., 2026. Massive parallel deep reinforcement learning for active slam. URL: https://arxiv.org/abs/2603.25834
Macenski, S., Foote, T., Gerkey, B., Lalancette, C., Woodall, W., 2022. Robot operating system 2: Design, architecture, and uses in the wild. Science Robotics 7 (66), eabm6074. URL: https://www.science.org/doi/abs/10.1126/scirobotics.abm6074 DOI: 10.1126/scirobotics.abm6074
Macenski, S., Jambrecic, I., 2021. Slam toolbox: Slam for the dynamic world. Journal of Open Source Software 6 (61), 2783. URL: https://doi.org/10.21105/joss.02783 DOI: 10.21105/joss.02783
Niroui, F., Zhang, K., Kashino, Z., Nejat, G., 2019. Deep reinforcement learning robot for search and rescue applications: Exploration in unknown cluttered environments. IEEE Robotics and Automation Letters 4 (2), 610–617. DOI: 10.1109/LRA.2019.2891991
Placed, J. A., Castellanos, J. A., 2020. A deep reinforcement learning approach for active slam. Applied Sciences 10 (23). URL: https://www.mdpi.com/2076-3417/10/23/8386 DOI: 10.3390/app10238386
Sutton, R. S., Barto, A. G., 2018. Reinforcement Learning: An Introduction, 2nd Edition. The MIT Press. URL: http://incompleteideas.net/book/the-book-2nd.html
Thrun, S., Burgard, W., Fox, D., 2005. Probabilistic robotics. MIT Press, Cambridge, Mass.
Topiwala, A., Inani, P., Kathpal, A., 2018. Frontier based exploration for autonomous robot. URL: https://arxiv.org/abs/1806.03581
Wang, R., Zhang, J., Lyu, M., Yan, C., Chen, Y., 2024. An improved frontierbased robot exploration strategy combined with deep reinforcement learning. Robotics and Autonomous Systems 181, 104783. URL: https://www.sciencedirect.com/science/article/pii/S0921889024001672 DOI: https://doi.org/10.1016/j.robot.2024.104783
Zhao, S., Hwang, S.-H., 2024. Exploration- and exploitation-driven deep deterministic policy gradient for active slam in unknown indoor environments. Electronics 13 (5). URL: https://www.mdpi.com/2079-9292/13/5/999 DOI: 10.3390/electronics13050999
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Javier Arambarri-Calvo, Raul Fernandez-Fernandez, Jesús Chacón
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
