Estimating Probability Distribution with Q-learning for Biped Gait Generation and Optimization

A new biped gait generation and optimization method is proposed in the frame of estimation of distribution algorithms (EDAs) with Q-learning method. By formulating the biped gait synthesis as a constrained multi-objective optimization problem, a dynamically stable and low energy cost biped gait is generated by EDAs with Q-learning (EDA_Q), which estimate probability distributions derived from the objective function to be optimized to generate searching points in the highly-coupled and high dimensional working space of biped robots. To get the preferable permutation of the interrelated parameters, Q-learning is combined to build and modify the probability models in EDA autonomously. By making use of the global optimization capability of EDA, the proposed EDA_Q can also solve the local minima problem in traditional Q-learning. On the other hand, with the learning agent, EDA_Q can evaluate the probability distribution model regularly without pre-designed structure and updating rule. The simulation results show that faster and more accurate searching can be achieved to generate preferable biped gait. The gait has been successfully used to drive a soccer-playing humanoid robot called Robo-Erectus which is one of the foremost leading soccer-playing humanoid robots in the RoboCup Humanoid League