Complex event processing in supercomputer environment sensor and neural network based analysis

Large number of sensors are running in the high performance computer (HPC, supercomputer) environment to detect the hardware and software resource utilization during the jobs execution. Different state variables of the HPC system are detected by physical sensors but virtual sensors are producing important event information, as well. By measuring sensor data of different processes huge amount of state information is collected from the supercomputer during the execution time. Each of the time series collected in this way have special meaning and characterize the utilization of a given HPC resource subset. Having dozens of time series collected in parallel for each of the 128 computation nodes, complex event processing (CSP) methods are needed to evaluate the jobs execution efficiency. This paper gives a complex overview of a resource utilization analysis of 18 TFLOPS capacity HPC system with 1.5 thousand CPU cores. Advanced method is presented based on neural networks for defining clusters of the strongly correlated variables with 55 thousand samples each. Selection procedure of the representing variable needed for each cluster. Method based on statistical analysis is proposed to evaluate jobs execution efficiency by the HPC system.