Using reinforcement learning to catch a baseball

Moments after a baseball batter has hit a fly ball, an outfielder has to decide whether to run forward or backward to catch the ball. Judging a fly ball is a difficult task, especially when the fielder is in the plane of the ball´s trajectory. A previous study in experimental psychology suggests that to intercept the ball, the fielder has to run such that d²(tanφ)/dt² is close to zero, where φ is the elevation angle of the ball from the fielder´s perspective. The authors investigate whether d²(tanφ)/dt² information is sufficient to learn this task in two reinforcement learning models: AHC and Q learning. The authors´ results indicate that although d²(tanφ)/dt² provides initial clue as to the ball´s landing point, it is not a good indicator in the latter stages of the ball´s trajectory. Thus the two models fail to learn to intercept fly balls. However, when information about the perpendicular velocity of the ball with respect to the fielder is also included as an input to the system, it provides the necessary discriminability in the latter stages of the ball´s trajectory, and the two models are able to successfully learn this reinforcement problem