Automated testing in the continuous delivery pipeline: A case study of an online company

Companies running an online business need to be able to frequently push new features and bug fixes from development into production. Successful high-performance online companies deliver code changes often several times per day. Their continuous delivery model supports the business needs of the online world. At the same time, however, such practices increase the risk of introducing quality issues and unwanted side effects. Rigorous test automation is therefore a key success factor for continuous delivery. In this paper we describe how automated testing is used in the continuous delivery pipeline of an Austrian online business company. The paper illustrates the complex technical and organizational challenges involved and summarizes the lessons from more than six years of practical experience in establishing and maintaining an effective continuous delivery pipeline.