Automatic Specification-Based Testing: Challenges and Possibilities

Automatic specification-based testing has long been a goal of software engineering, but due to various challenges and difficulties, the goal still remains far from realized. If the specification is informal, full automation would be impossible in general. Formal specification improves the situation, but there are many barriers lying ahead. Automatic test set generation can be difficult to meet high standard, but test result analysis for determining the existence of bugs and debugging for locating the bugs and deciding their nature seem to be even more challenging. This paper systematically discusses all of the major challenges in automatic specification-based testing, and describes some possibilities to deal with the challenges. Some open but interesting problems are presented to inspire future research in the field.