Recently, adiabatic quantum computing has been successfully used in machine learning tasks. One specific type of machine learning, Reinforcement Learning (RL), usually runs efficiently on classical computers.
Image credit: Gam Ol via Pexels (Free Pexels licence)
Nevertheless, as applications get more and more complex, quantum RL algorithms may also be beneficial. A type of stochastic neural networks, called Boltzmann machines, has been successfully combined with RL using quantum annealing. However, the method worked for small single-agent environments only.
Hence, a recent paper proposes a novel architecture to enable more complex domains and stabilize learning by using experience replay buffer and separating policy and target networks. This way, researchers are able to stabilize learning. While the architecture is still limited to relatively small domains, the needed quantum processing unit size may be reduced compared to previous approaches.
Reinforcement learning has driven impressive advances in machine learning. Simultaneously, quantum-enhanced machine learning algorithms using quantum annealing underlie heavy developments. Recently, a multi-agent reinforcement learning (MARL) architecture combining both paradigms has been proposed. This novel algorithm, which utilizes Quantum Boltzmann Machines (QBMs) for Q-value approximation has outperformed regular deep reinforcement learning in terms of time-steps needed to converge. However, this algorithm was restricted to single-agent and small 2×2 multi-agent grid domains. In this work, we propose an extension to the original concept in order to solve more challenging problems. Similar to classic DQNs, we add an experience replay buffer and use different networks for approximating the target and policy values. The experimental results show that learning becomes more stable and enables agents to find optimal policies in grid-domains with higher complexity. Additionally, we assess how parameter sharing influences the agents behavior in multi-agent domains. Quantum sampling proves to be a promising method for reinforcement learning tasks, but is currently limited by the QPU size and therefore by the size of the input and Boltzmann machine.
Research paper: Müller, T., Roch, C., Schmid, K., and Altmann, P., “Towards Multi-Agent Reinforcement Learning using Quantum Boltzmann Machines”, 2021. Link: https://arxiv.org/abs/2109.10900
