Localization and field test of a Dutch train time table information system

We are developing a system for providing information on train schedules in the Netherlands over the telephone. It is based on a continuous speech recognizer, an “AI-style” natural language processing module, and a dialogue manager that allows for considerable user initiative. The system is a Dutch localized version of an existing German prototype. In the localization process we have come across a problem that did not surface in the original German version, but that seems to be quite common in a number of other applications: in the Dutch railway time table days run from 4 am on one day to 3:59 am on the next day. This leads to discrepancies between the date of the computer running the application and the date system of the application itself. Interactive, mixed-initiative information systems are notoriously difficult to test objectively. For our system we have designed two such tests, a factory acceptance test (FAT) and a user acceptance test (UAT). In the FAT is was established whether the Dutch system fulfilled its design specifications with respect to the proportion of successfully completed queries, the system´s stability, speed, etc. Our system passed the test, but a number of issues about the test itself remain unresolved. The paper discusses approaches towards a resolution of these issues. The UAT is designed primarily as a means for getting information about the relative priority of aspects of the system that must be improved to enable full commercial deployment. The paper describes the final design of the UAT