Resource constrained design of artificial neural networks

The authors address the problem of automating the design of artificial neural networks based on a gradient-descent learning algorithm for solving a given application. There are many possible network configurations for solving the given application, and enumerating and training all of them is impossible in any reasonable amount of time. The authors present a heuristic design method for selecting and training promising neural networks with the goal of maximizing a given objective function of cost and training time and satisfying given resource constraints. The proposed method consists of two parts. For a given amount of time divided into quanta, the first part of the method systematically selects promising network configurations and trains them for various quanta of time. Unpromising configurations are pruned as the search proceeds. The second part of the method refines the error measure used in the first part. When the time limit is exceeded, a set of promising but possibly partially trained networks is reported. The authors have applied the method to a variety of benchmark problems and found it to be effective in identifying promising network configurations. Experimental results on the encode-encode and two-spiral problems illustrate that the proposed design method can be applied across different problems and that it scales well for problems of different sizes