No Access

A Novel Adaptive Sampling Strategy for Deep Reinforcement Learning

Xingxing Liang

National University of Defense Technology, College of Systems Engineering, Changsha, P. R. China

E-mail Address: doublestar_l@163.com

Xingxing Liang and Li Chen are co-first author.

Search for more papers by this author

Li Chen

National University of Defense Technology, College of Systems Engineering, Changsha, P. R. China

E-mail Address: chenli18@nudt.edu.cn

Xingxing Liang and Li Chen are co-first author.

Search for more papers by this author

Yanghe Feng

National University of Defense Technology, College of Systems Engineering, Changsha, P. R. China

E-mail Address: fengyanghe@nudt.edu.cn

Corresponding author.

Search for more papers by this author

Zhong Liu

National University of Defense Technology, College of Systems Engineering, Changsha, P. R. China

E-mail Address: phillipliu@263.net

Search for more papers by this author

Yang Ma

National University of Defense Technology, College of Systems Engineering, Changsha, P. R. China

E-mail Address: yang_ma_cn@163.com

Search for more papers by this author

, and

Kuihua Huang

National University of Defense Technology, College of Systems Engineering, Changsha, P. R. China

E-mail Address: kuihh@163.com

Search for more papers by this author

https://doi.org/10.1142/S1469026821500115Cited by:2 (Source: Crossref)

Abstract

Reinforcement learning, as an effective method to solve complex sequential decision-making problems, plays an important role in areas such as intelligent decision-making and behavioral cognition. It is well known that the sample experience replay mechanism contributes to the development of current deep reinforcement learning by reusing past samples to improve the efficiency of samples. However, the existing priority experience replay mechanism changes the sample distribution in the sample set due to the higher sampling frequency assigned to a specific transition, and it cannot be applied to actor-critic and other on-policy reinforcement learning algorithm. To address this, we propose an adaptive factor based on TD-error, which further increases sample utilization by giving more attention weight to samples of larger TD-error, and embeds it flexibly into the original Deep Q Network and Advantage Actor-Critic algorithm to improve their performance. Then we carried out the performance evaluation for the proposed architecture in the context of CartPole-V1 and 6 environments of Atari game experiments, respectively, and the obtained results either on the conditions of fixed temperature or annealing temperature, when compared to those produced by the vanilla DQN and original A2C, highlight the advantages in cumulative rewards and climb speed of the improved algorithms.

Keywords:

Remember to check out the Most Cited Articles!
Check out these titles in artificial intelligence!

References

1. V. Mnih, K. Kavukcuoglu, D. Silver, A. Rusu, J. Veness, M. G. Bellemare et al., Human-level control through deep reinforcement learning, Nature 518(7540) (2015) 529–533. Crossref, Google Scholar
2. D. Silver, A. Huang, C. J. Maddison et al., Mastering the game of Go with deep neural networks and tree search, Nature 529(7587) (2016) 484–489. Crossref, Google Scholar
3. D. Silver, J. Schrittwieser, K. Simonyan et al., Mastering the game of Go without human knowledge, Nature 550(7676) (2017) 354–359. Crossref, Google Scholar
4. https://blog.openai.com/openai-five/. Google Scholar
5. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd edn. (inpreparation) (MIT Press, 2017). Google Scholar
6. M. Volodymyr, P. B. Adrià, M. Mehdi et al., Asynchronous methods for deep reinforcement learning, in Proc. 33rd Int. Conf. Machine Learning (ACM, New York City, NY, US, 2016), pp. 2850–2869. Google Scholar
7. A. W. Moore and C. G. Atkeson, Prioritized sweeping: Reinforcement learning with less data and less time, Mach. Learn. 13(1) (1993) 103–130. Crossref, Google Scholar
8. D. Andre, N. Friedman and R. Parr, Generalized prioritized sweeping, in Proc. 11th Annual Conf. Neural Information Processing Systems (ACM, Denver, CO, US, 1997), pp. 1001–1007. Google Scholar
9. A. White, J. Modayil and R. S. Sutton, Surprise and curiosity for big data robotics, in Proc. 28th AAAI Conf. Artificial Intelligence (ACM, Quebec City, QC, Canada, 2014), pp. 19–23. Google Scholar
10. D. Horgan, J. Quan, D. Budden et al., Distributed prioritized experience replay, in Proc. 6th Int. Conf. Learning Representations (ICLR, Vancouver, BC, Canada, 2018). Google Scholar
11. Z. Wang, T. Schaul, M. Hessel et al., Dueling network architectures for deep reinforcement learning, in Proc. 33rd Int. Conf. Machine Learning (ACM, New York City, NY, US, 2016), pp. 2939–2947. Google Scholar
12. M. Jaderberg, V. Mnih, W. M. Czarnecki et al., Reinforcement learning with unsupervised auxiliary tasks, in Proc. 5th Int. Conf. Learning Representations (ICLR, Toulon, France, 2017). Google Scholar
13. T. Hester, T. Schaul, A. Sendonaris et al., Deep q-learning from demonstrations, in Proc. 32nd AAAI Conf. Artificial Intelligence (ACM, New Orleans, LA, US, 2018), pp. 3223–3230. Crossref, Google Scholar
14. M. Hessel, J. Modayil, H. V. Hasselt et al., Rainbow: Combining improvements in deep reinforcement learning, in Proc. 32nd AAAI Conf. Artificial Intelligence (ACM, New Orleans, LA, US, 2018), pp. 3215–3222. Crossref, Google Scholar
15. T. Pohlen, B. Piot, T. Hester et al., Observe and Look Further: Achieving Consistent Performance on Atari, arXiv:1805.11593. Google Scholar
16. H. Dan, J. Quan, D. Budden et al., Distributed prioritized experience replay, in Proc. 5th Int. Conf. Learning Representations (ICLR, Vancouver, BC, Canada, 2018). Google Scholar
17. L. Matignon, G. J. Laurent and N. L. Fort-Piat, Hysteretic q-learning: An algorithm for decentralized reinforcement learning in cooperative multi-agent teams, in Proc. the 2007 IEEE/RSJ Int. Conf. Intelligent Robots and Systems (IEEE, San Diego, CA, US, 2007), pp. 64–69. Crossref, Google Scholar
18. M. A. Potter and K. A. D. Jong, A cooperative coevolutionary approach to function optimization, Third Parallel Problem Solving Form Nature 866 (1994) 249–257. Crossref, Google Scholar
19. Y. Xu, W. Zhang, W. Liu et al., Multiagent-based reinforcement learning for optimal reactive power dispatch, IEEE Trans. Syst. Man Cybernet. Part C Appl. Rev. 42(6) (2012) 1742–1751. Crossref, Google Scholar
20. L. Matignon, G. J. Laurent and N. L. Fort-Piat, Independent reinforcement learners in cooperative Markov games: a survey regarding coordination problems, Knowl. Eng. Rev. 27(1) (2012) 1–31. Crossref, Google Scholar
21. N. Barbalios and P. Tzionas, A Robust Approach for Multi-agent Natural Resource Allocation based on Stochastic Optimization Algorithms (Elsevier Science Publishers B. V., 2014). Crossref, Google Scholar
22. S. Omidshafiei, J. Pazis, C. Amato et al., Deep decentralized multi-task multi-agent reinforcement learning under partial observability, in Proc. 34rd Int. Conf. Machine Learning (ACM, Sydney, NSW, Australia, 2017), pp. 4108–4122. Google Scholar
23. E. Wei and S. Luke, Lenient learning in independent-learner stochastic cooperative games, J. Machine Learn. Res. 17 (2016) 2914–2955. Google Scholar
24. G. Palmer, K. Tuyls, D. Bloembergen et al., Lenient multi-agent deep reinforcement learning, in Proc. 17th Int. Conf. Autonomous Agents and Multiagent Systems (IFAAMAS, Stockholm, Sweden, 2018), pp. 443–451. Google Scholar