A new approach to fault-tolerant wormhole routing for mesh-connected parallel computers

A new method for fault-tolerant routing in arbitrary dimensional meshes is introduced. The method was motivated by certain routing requirements of an initial design of the Blue Gene supercomputer project currently underway in IBM Research. Among the requirements were to provide deterministic deadlock free wormhole routing in a 3-dimensional mesh, in the presence of many faults (up to a few percent of the many thousands of nodes in the machine), while using two virtual channels. It was also desired to minimize the number of "turns" in each route, i.e., the number of times that the route changes direction. There has been much work on routing methods for meshes that route messages around faults or regions of faults. The new method is to declare certain good nodes to be "lambs": a lamb is used for routing but not processing, so a lamb is neither the source nor the destination of a message. The lambs are chosen so that every "survivor node", a node that is neither faulty nor a lamb, can reach every survivor node by at most two rounds of dimension-ordered (such as e-cube) routing. An algorithm for finding a set of lambs is presented The results of simulations on 2D and 3D meshes of various sizes with various numbers of random node faults are given. For example, on a 32×32×32 3D mesh with 3% random faults, and using two rounds of e-cube routing for each message, the average number of lambs is less than 68. which is less than 7% of the number 983 of faults. The computational complexity offending the minimum number of lambs for a given fault set is also explored, and this problem is shown to be NP-hard fir 3-dimensional meshes with two rounds of e-cube routing.