Robotic Manipulation Using Inverse Reinforcement Learning with Expert-Trajectory-Based Features
DOI:
https://doi.org/10.17979/ja-cea.2025.46.12175Keywords:
Machine Learning, Robotic Manipulation, Deep Reinforcement Learning, Inverse Reinforcement LearningAbstract
Algorithms for robotic manipulation learning still face challenges in tasks with high variability and dimensionality. Among
them, Reinforcement Learning has shown good results but is limited by the definition of the reward function. This has led to the
development of Inverse Reinforcement Learning (IRL) algorithms, which estimate the reward from expert demonstrations. This
work proposes and validates an IRL approach based on expert-trayectory based features, applied to manipulation tasks with the
TIAGo++ robot. The method leverages demonstrations to focus the feature definition on the relevant regions of the state space
for the expert, as well as to prioritize final states close to the goal. The selected manipulation tasks were block stacking and
cabinet opening. They were trained in simulation and transferred to the real robot, demonstrating the viability and effectiveness
of the approach both in successful execution and in distance-based metrics relative to the expert.
References
Abbeel, P., Ng, A. Y., 2004. Apprenticeship learning via inverse reinforcement learning. Proceedings, Twenty-First International Conference on Machine Learning, ICML 2004, 1–8. DOI: 10.1145/1015330.1015430
Arora, G., KiranBala, Emadifar, H., Khademi, M., 11 2023. A review of radial basis function with applications explored. Journal of the Egyptian Mathe-matical Society 2023 31:1 31, 1–14. DOI: 10.1186/S42787-023-00164-3
Ashwood, Z. C., Jha, A., Pillow, J. W., 12 2022. Dynamic inverse reinfor-cement learning for characterizing animal behavior. Advances in Neural Information Processing Systems 35, 29663–29676.
Beliaev, M., Pedarsani, R., 2 2024. Inverse reinforcement learning by estimating expertise of demonstrators. DOI: 10.1609/aaai.v39i15.33705
Bellman, R., 1961. Adaptive Control Processes: A Guided Tour. Princeton University Press.
Bigham, B. S., Mazaheri, S., 2020. A survey on measurement metrics for shape matching based on similarity, scaling and spatial distance. Lecture Notes on Data Engineering and Communications Technologies 45, 13–23. DOI: 10.1007/978-3-030-37309-2 2
Eschmann, J., 2021. Reward function design in reinforcement learning. Studies in Computational Intelligence 883, 25–33. DOI: 10.1007/978-3-030-41188-6 3 /FIGURES /3
Fu, J., Luo, K., Levine, S., 10 2017. Learning robust rewards with adversarial inverse reinforcement learning. 6th International Conference on Learning Representations, ICLR 2018 - Conference Track Proceedings. DOI: 10.48550/arXiv.1710.11248
Hoshino, H., Ota, K., Kanezaki, A., Yokota, R., 2022. Opirl: Sample efficient off-policy inverse reinforcement learning via distribution matching. Proceedings - IEEE International Conference on Robotics and Automation, 448-454. DOI: 10.1109/ICRA46639.2022.9811660
Jain, A. K., 6 2010. Data clustering: 50 years beyond k-means. Pattern Recognition Letters 31, 651–666. DOI: 10.1016/J.PATREC.2009.09.011
Jin, M., Damianouy, A., Abbeel, P., Spanos, C., 12 2017. Inverse reinforcement learning via deep gaussian process. Uncertainty in Artificial Intelligence - Proceedings of the 33rd Conference, UAI 2017. DOI: 10.48550/arXiv.1512.08065
Kober, J., Bagnell, J. A., Peters, J., 9 2013. Reinforcement learning in robotics: A survey. International Journal of Robotics Research 32, 1238–1274. DOI: 10.1177/0278364913495721
Likmeta, A., Metelli, A. M., Ramponi, G., Tirinzoni, A., Giuliani, M., Restelli, M., 9 2021. Dealing with multiple experts and non-stationarity in inverse reinforcement learning: an application to real-life problems. Machine Learning 110, 2541–2576. DOI: 10.1007/S10994-020-05939-8/FIGURES/20
Naranjo-Campos, F. J., Victores, J. G., Balaguer, C., 2024. Expert-trajectory-based features for apprenticeship learning via inverse reinforcement learning for robotic manipulation. Applied Sciences 14 (23), 11131. DOI: 10.3390/app142311131
Neu, G., Szepesvári, C., 6 2012. Apprenticeship learning using inverse reinforcement learning and gradient methods. Proceedings of the 23rd Conference on Uncertainty in Artificial Intelligence, UAI 2007, 295–302. DOI: 10.48550/arXiv.1206.5264
Ng, A. Y., Russell, S., 2000. Algorithms for inverse reinforcement learning.
Poiani, R., Curti, G., Metelli, A. M., Restelli, M., 2024. Inverse reinforcement learning with sub-optimal experts. DOI: 10.48550/arXiv.2401.03857
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Openai, O. K., 7 2017. Proximal policy optimization algorithms. DOI: 10.48550/arXiv.1707.06347
Sun, J., Yu, L., Dong, P., Lu, B., Zhou, B., 4 2021. Adversarial inverse reinforcement learning with self-attention dynamics model. IEEE Robotics and Automation Letters 6, 1880–1886. DOI: 10.1109/LRA.2021.3061397
Sutton, R. S., Barto, A. G., 2018. Reinforcement Learning: An Introduction, 2nd Edition. MIT press.
Wulfmeier, M., Ondrúška, P., Ondrúška, O., Posner, I., 7 2015. Maximum entropy deep inverse reinforcement learning. DOI: 10.48550/arXiv.1507.04888
Zeng, S., Li, C., Garcia, A., Hong, M., 2 2023. When demonstrations meet generative world models: A maximum likelihood framework for offline inverse reinforcement learning.
Ziebart, B. D., Maas, A., Bagnell, J. A., Dey, A. K., 2008. Maximum entropy inverse reinforcement learning. In: Proceedings of the Twenty-Third AAAI Conference on Artificial Intelligence.
Downloads
Published
Issue
Section
License
Copyright (c) 2025 Francisco J. Naranjo-Campos, Juan G. Victores, Ana Calzada-García, Carlos Balaguer
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
