Vision-based deep reinforcement learning to control a manipulator

Abstract

In this paper, we apply Reinforcement Learning (RL) to control a manipulator using camera images. Basically, RL algorithm helps the agent to choose actions under some specific policies, which maximize rewards accumulated from a sequential result of the actions. In order to control a manipulator, the agent generally considers joint angles or torques of the manipulator as the states. Deep learning has enabled to deal with a very high dimensional state space so that raw images can be considered as states. However, using raw images as state spaces can require extensive computational resources because of their high dimension, so there has been much research using additory algorithms such as Convolutional Neural Network (CNN) or autoencoder to extract features from images. While those algorithms reduce the computational load, it is still a quite complicated process. In this paper we consider a task for the end-effector to reach a random target and propose a new approach using a vision-based direction vector, which has low dimension and can be simply implemented. We calculate the direction vectors via the camera and exploit them as the states of the policy. Then, based on the direction vectors, the manipulator can learn how to control each joint. For the simulation, we demonstrate a multiple degree of freedom(dof) manipulator of a commercial robot, and the simulation results show that the manipulator successfully tracks the target.