Framework to Support Scenario Development for Human-Centered Alerting System Evaluation

The purpose of the framework introduced here is to support the development of evaluation scenarios that are capable of assessing system level performance while considering the system, the humans that interact with it, and the environment. The following five step framework is presented and applied to a pilot self separation task: 1) identify entities critical to system design, development, and operation and define their goals and properties as they relate to the system being studied; 2) define a subset of functionality for evaluation (define an execution sequence); 3) map entity properties to the execution sequence to identify independent variables; 4) translate entity goals into a set of system goals that can be used to identify dependent measures; and 5) iterate through each step to ensure the models produced are internally consistent.