Performance and operational experience with the heterogeneous farm of the ATLAS trigger and data acquisition system

The ATLAS trigger and data acquisition (TDAQ) is a distributed, multi trigger level, data-acquisition system, mostly made of off-the-shelf processing units organized in a farm. In its final configuration the system will account more than 2000 nodes, having heterogeneous capabilities and network connections, due to the TDAQ program for rolling expansions and upgrades. In this paper we present how we dealt with the farm heterogeneity during the proton-proton collisions of 2010 and 2011: a period characterized by changing working conditions, and constantly increasing LHC instantaneous luminosity. We describe a graphical tool to balance the computing-power and bandwidth sharing across the trigger farms, a data-flow monitoring daemon that provides high-level resource-aware data-flow operational information and the evolution of data-flow communication protocols.