Journal of the Korea Institute of Military Science and Technology (한국군사과학기술학회지)

The Korea Institute of Military Science and Technology (한국군사과학기술학회)
권호 표지
DOI QR코드

DOI QR Code

Roll control of Underwater Vehicle based Reinforcement Learning using Advantage Actor-Critic

Advantage Actor-Critic 강화학습 기반 수중운동체의 롤 제어

  • Lee, Byungjun (The 4th Research and Development Institute, Agency for Defense Development)
  • 이병준 (국방과학연구소 제4기술연구본부)
  • Received : 2020.09.18
  • Accepted : 2021.01.22
  • Published : 2021.02.05
Download PDF
⟨ Previous Next ⟩

Abstract

In order for the underwater vehicle to perform various tasks, it is important to control the depth, course, and roll of the underwater vehicle. To design such a controller, it is necessary to construct a dynamic model of the underwater vehicle and select the appropriate hydrodynamic coefficients. For the controller design, since the dynamic model is linearized assuming a limited operating range, the control performance in the steady state is well satisfied, but the control performance in the transient state may be unstable. In this paper, in order to overcome the problems of the existing controller design, we propose a A2C(Advantage Actor-Critic) based roll controller for underwater vehicle with stable learning performance in a continuous space among reinforcement learning methods that can be learned through rewards for actions. The performance of the proposed A2C based roll controller is verified through simulation and compared with PID and Dueling DDQN based roll controllers.

Keywords

References

  1. Jongho Shin, and Sanghyun Joo, "NN-based Adaptive Control for a Skid-Type Autonomous Unmanned Ground Vehicle," Journal of Institute of Control, Robotics and Systems, Vol. 20, No. 12, pp. 1278-1283, 2014. https://doi.org/10.5302/J.ICROS.2014.14.8023
  2. S. Y. Kim, et. al., "Neural Network for a Roll Control of the Underwater Vehicle," KIMST Annual Conference Proceedings, pp. 14-15, 2018.
  3. H.-J. Chae, et. al., "Time-varying Proportional Navigation Guidance using Deep Reinforcement Learning," Journal of the Korea Institute of Military Science and Technology, Vol, 23, No. 4, pp. 399-406, 2020. https://doi.org/10.9766/KIMST.2020.23.4.399
  4. S. Y. Kim, et. al., "Reinforcement Learning for a Roll Control of the Unmanned Underwater Vehicle," Naval Ship Technology & Weapon Systems Seminar Proceedings, pp. 474-477, 2019.
  5. Volodymyr Mnih, et. al., "Playing Atari with Deep Reinforcement Learning," In NIPS Deep Learning Workshop, 2013.
  6. Hado van Hasselt, et. al., "Deep Reinforcement Learning with Double Q-learning," AAAI, Vol. 16, 2016.
  7. Ziyu Wang, et. al., "Dueling Network Architectures for Deep Reinforcement Learning," Proceedings of The 33rd International Conference on Machine Learning, 2016.
  8. R. S. Sutton, and A. G. Barto, "Reinforcement Learning: An Introduction," The MIT Press, pp. 328-333, 2018.
  9. W. W. Lee, et. al., "Reinforcement Learning with Python and Keras," Wikibook, pp. 225-277, 2020.
  10. H. J. Cho, et. al., "A Two-Stage Initial Alignment Technique for Underwater Vehicles Dropped from a Mother Ship," International Journal of Precision Engineering and Manufacturing, Vol. 14, No. 12, pp. 2067-2073, 2013. https://doi.org/10.1007/s12541-013-0280-y
  11. Arun Nair, et. al., "Massively Parallel Methods for Deep Reinforcement Learning," In ICML Deep Learning Workshop, 2015.
  12. S. Y. Kim, et. al., "Robust Depth Control for an Autonomous Navigation of the Underwater Vehicle," KIMST Annual Conference Proceedings, pp. 14-15, 2013.
  13. Benjamin C. Kuo, "Automatic Control Systems," Prentice Hall, 1994.