Test Instrumentation and Pattern Matching for Automatic Failure Identification

An increasing emphasis on test automation presents test teams with a growing set of results to review and investigate. The life time of a given failure typically spans multiple automation runs making the necessary due-diligence effort time consuming, labor intensive, error prone and often redundant. When faced with this condition within their teams, the authors addressed the problem from two approaches; pattern recognition and instrumentation. Pattern recognition sought to automate some of the manual processes of failure identification while instrumentation provided a detailed and consistent description of a failure. In practice, the problem divides into 3 domains; instrumentation, annotation, and recognition. Instrumentation forms the basis for the solution. Annotation provides the ability to refine a given pattern based on the investigation results. Recognition forms the basis for identification. The effectiveness of the solution in a production environment will be discussed. Best practices and lessons learned will be shared.