The ATLAS High Level Trigger infrastructure, performance and future developments

The ATLAS High Level Trigger (HLT) is a distributed real-time software system that performs the final online selection of events produced during proton-proton collisions at the Large Hadron Collider (LHC). It is designed as a two-stage event filter running on a farm of commodity PC hardware. Currently the system consists of about 850 multi-core processing nodes that will be extended incrementally following the increasing luminosity of the LHC to about 2000 nodes depending on the evolution of the processor technology. Due to the complexity and similarity of the algorithms a large fraction of the software is shared between the online and offline event reconstruction. The HLT Infrastructure serves as the interface between the two domains and provides common services for the trigger algorithms. The consequences of this design choice will be discussed and experiences from the operation of the ATLAS HLT during cosmic ray data taking and first beam in 2008 will be presented. Since the event processing time at the HLT is directly related to the number of processing nodes required, special emphasis has to be put on monitoring and improving the performance of the software. Both open-source as well as custom developed tools are used for this task and a few use-cases will be shown. Finally, the implications of the prevailing industry trend towards multi- and many-core processors for the architecture of the ATLAS HLT will be discussed. The use of multi-processing and multi-threading techniques within the current system will be presented. Several approaches to make optimal use of the increasing number of cores will be investigated and the practical implications of implementing each approach in the current system with hundreds of developers and several hundred thousand lines of code will be examined.