Journal of Korea Game Society (한국게임학회 논문지)

Korea Game Society (한국게임학회)
권호 표지
DOI QR코드

DOI QR Code

A Study about the Usefulness of Reinforcement Learning in Business Simulation Games using PPO Algorithm

경영 시뮬레이션 게임에서 PPO 알고리즘을 적용한 강화학습의 유용성에 관한 연구

  • 양의홍 (홍익대학교 일반대학교 게임학과(공학)) ;
  • 강신진 (홍익대학교 일반대학교 게임학과(공학)) ;
  • 조성현 (홍익대학교 일반대학교 게임학과(공학))
  • Received : 2019.09.10
  • Accepted : 2019.11.08
  • Published : 2019.12.20
Download PDF
⟨ Previous Next ⟩

Abstract

In this paper, we apply reinforcement learning in the field of management simulation game to check whether game agents achieve autonomously given goal. In this system, we apply PPO (Proximal Policy Optimization) algorithm in the Unity Machine Learning (ML) Agent environment and the game agent is designed to automatically find a way to play. Five game scenario simulation experiments were conducted to verify their usefulness. As a result, it was confirmed that the game agent achieves the goal through learning despite the change of environment variables in the game.

본 논문에서는 경영 시뮬레이션 게임 분야에서 강화학습을 적용하여 게임 에이전트들이 자율적으로 주어진 목표를 달성하는지를 확인하고자 한다. 본 시스템에서는 Unity Machine Learning (ML) Agent 환경에서 PPO (Proximal Policy Optimization) 알고리즘을 적용하여 게임 에이전트가 목표를 달성하기 위해 자동으로 플레이 방법을 찾도록 설계하였다. 그 유용성을 확인하기 위하여 5가지의 게임 시나리오 시뮬레이션 실험을 수행하였다. 그 결과 게임 에이전트가 다양한 게임 내 환경 변수의 변화에도 학습을 통하여 목표를 달성한다는 것을 확인하였다.

Keywords

References

  1. Sungpill Kim, Deep Learning First Step, pp.17-33, Hanbit Media, 2016.
  2. Taewoo Lee, Jinhoo Ryu, Heemin Park "Hovering Control of 1-Axial Drone with Reinforcement Learning", Journal of Korea Multimedia Society, Vol.21, No.2, pp.250-260, 2018. https://doi.org/10.9717/kmms.2018.21.2.250
  3. Daniel R.Jiang, Emmanuel Ekwedike, Han Liu, "Feedback-Based Tree Search for Reinforcement Learning", Journal of Korea Multimedia Society, arXiv:1805.05935, 2018.
  4. Jeongsoo Han, "A Study of Adaptive QoS Routing scheme using Policy-gradient Reinforcement Learning", Journal of the Korea Society of Computer and Information, Vol.16, No.2, pp.93-99, 2011. https://doi.org/10.9708/jksci.2011.16.2.093
  5. Jongho Kim, Daesung Kang, Jooyoung Park, "Robot Locomotion via RLS-based Actor-Critic Learning", Journal of Korean Institute of Intelligent Systems, Vol.15, No.7, pp.893-898, 2005. https://doi.org/10.5391/JKIIS.2005.15.7.893
  6. Arthur Juliani, "Introducing: Unity Machine Learning Agents Toolkit", Unity Blog, https://blogs.unity3d.com/2017/09/19/introducin g-unity-machine-learning-agents/, 2017.
  7. Wooil Shim, Taehwa Park, Kyungjoong Kim, "Comparison of Policy Optimization Reinforcement Learning for Simulated Autonomous Car Environment", Korea Information Science Society, p.833-835, 2018.
  8. Adrian Gonzalez, Ramirez, "Neural networks applied to a tower defense video game", Universitat Jaume I, Grauen Disseny i Desenvolupament de Videojocs [94], 2018.
  9. Arthur Juliani, Vincent-Pierre Berges, Esh Vckay, Yuan Gao, Hunter Henry, Marwan Mattar, Danny Lange, "ML-Agents Toolkit Overview",https://github.com/Unity-Technologies/ml-agents/blob/master/docs/ML-Agents-Overview.md, 2017.
  10. Jaehoon Lee, Taerim Kim, Jonggyu Song, Hyunjae Im, "Flight Trajectory Simulation via Reinforcement Learning in Virtual Environment", Journal of the Korea Society for Simulation, Vol.27, No.4, p.1-8, 2018. https://doi.org/10.9709/JKSS.2018.27.4.001
  11. Sonic, "PPO (Proximal Policy Optimization Algorithms) I Machine Learning & QA)", Naver Blog, https://cafe.naver.com/soynature/2400, 2017.
  12. Saemaro Moon, Yonglak Choi "A Study on Application of Reinforcement Learning Algorithm Using Pixel Data", Journal of Information Technology Services, Vol.15, No.4, pp.85-95, 2016. https://doi.org/10.9716/KITS.2016.15.4.085
  13. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov, "Proximal Policy Optimization Algorithms", OpenAI, arxiv.org/pdf/1707.06347, 2017.
  14. RL Korea, "PG Travel Guide", RLKoreaBlog, https://reinforcement-learning-kr.github.io/2018/06/29/0_pg-travel-guide/#, 2018.
  15. Kyeongnam Kim, "ML-Agents Project Organization Unity ML / Unity", Naver Blog, https://blog.naver.com/kkyy0126/221448746477, 2019.
  16. Arthur Juliani, Vincent-Pierre Berges, Esh Vckay, Yuan Gao, Hunter Henry, Marwan Mattar, Danny Lange, "Training with Proximal Policy Optimization", https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Training-PPO.md, 2017.
  17. Arthur Juliani, Vincent-Pierre Berges, Esh Vckay, Yuan Gao, Hunter Henry, Marwan Mattar, Danny Lange, "Getting Started with the 3D Balance Ball Environment", https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Getting-Started-with-Balance-Ball.md#observing-training-progress, 2017.