Actor-Q based active perception learning system

An active perception learning system based on reinforcement learning is proposed. A novel reinforcement architecture, called Actor-Q, is employed in which Q-learning and actor-critic are combined. The system decides its actions according to Q-values. One of the actions is to move its sensor, and the others are to make an answer of its recognition result, each of which corresponds to each pattern. When the sensor motion is selected, the sensor moves according to the actor´s output signals. The Q-value for the sensor motion is trained by Q-learning, and the Actor is trained by the Q-value for the sensor motion on behalf of the critic. When one of the other actions is selected, the system outputs the recognition result. When the recognition answer is correct, the Q-value is trained to be the upper limit of the Q-value, and when the answer is not correct, it is trained to be 0.0. The module to compute Q-value and the actor module are both consisted of a neural network, and are trained by error backpropagation. The training signals are generated based on the above reinforcement learning. It was confirmed by some simulations using a visual sensor with nonuniform visual cells that the system moves its sensor to the place where it can recognize the presented pattern correctly. Even though the Q-value surface as a function of the sensor location has some local peaks, the sensor was not trapped and moved to the appropriate direction because the Q-value for the sensor motion becomes larger.