Monitoring High Performance Networks in Large-scale Clusters

The number of large-scale clusters is rising. They are included into grids or become key components of large structures. As more users and projects rely on RFC clusters, high availability and security are requirements for a fast growing adoption and use. In this paper, we, focus on high performance networks. All HPC clusters are built on top of them. We demonstrate that classical instrumentations are inefficient in HPC environment, they do not scale or cause a significant loss of performance. Based on this fact, we highlight clusters properties; nodes have assigned roles and are coupled at various levels. Moreover, we study the main characteristics of resource usage for each type of node and propose an instrumentation that can be effectively deployed. It results in fine-grained mechanisms adapted to system architecture, and performance constraints. Relevant information is collected over time. Two properties are verified online and dynamically: coherency and containment. Each induces a type of verification and both aim at reducing recovery time from failure and security risk of a whole cluster. We illustrate our methodology on QsNet by K. Magontis et al. (2001) network and provide a way to increase safety of high performance networks and clusters