Testing performance of IT service associates under variable load

One of the challenges in IT Service Management is an assessment of an innovation benefit introduced into service delivery on performance of IT System Administrator (SA). The question is especially complex when variability of the service requests load is taken into account. We used the sequential probability ratio test to assess effects of innovation for a case when a load of the service teams is constant over time. One of the drawbacks of this test is its requirement for a constant SAs team size while the arrival patterns of service requests might require otherwise. To address the volatility of a service request load we consider the possibility of changing number of service associates. We design a performance test based on the concepts of Repeated Significance Test (RST), which allowed us to take variable number of SAs into account. In this work we show the superiority of the suggested test in comparison to the fixed sample size test (FSST) in a number of experiments required to reach the definitive conclusion. We also compare RST with SPRT when the size of the service team is constant.