Model-free learning of wire winding control

In this paper we introduce a reinforcement learning approach to optimize the wire profile generated by an automated wire winding machine. The wire winder spools wire onto large bobbins, while trying to maintain an even wire profile across the bobbin. Uneven profiles that contain bumps or gaps (i.e. areas with too much or too little wire) lead to snagged or breaking wires when the bobbin is unwound. By setting the turning points of the traversal system which distributes the wire over a spinning bobbin, a controller can influence the amount of wire spooled on the edges of the bobbin. The behavior of the wire, however, is highly non-deterministic and difficult to model with sufficient accuracy, making the application of a model based controller technique very difficult. This fact makes reinforcement learning a promising approach to apply here, as this technique can learn optimal policies relying only on interactions with the plant. We apply a learning algorithm called continuous reinforcement learning automata and empirically demonstrate that this technique can successfully optimize the wire profile, even on rounded bobbins that require continuous adaptation of the turning point.