A framework for metrics in large complex systems

Terrorism information awareness (TIA) was initiated by the Information Awareness Office (IAO) of the Defense Advanced Research Projects Agency (DARPA) (Snowden and Kurtz, 2002; 2003; Snowden, 2000). TIA capabilities enable the detection, classification, identification, and tracking of terrorist activities in order to provide early warning of plans and to identify options to prevent them from being executed. TIA operated as a research program in an environment with real data, real users, and real missions. This environment presented many challenges to the collection and analysis of metrics. Like most large things, the TIA program was a mixture of the "complicated" and the "complex". At a high level, the program could be considered as both a "system" and a human "network". The "system" is complicated, able to be analyzed from its constituent components. On the other hand, the "network" is a complex composite of the activities of small groups engaged in problem solving. To analyze TIA experiments, we adopted multiple interwoven sets of "perspectives" encompassing both the system and the net. All perspectives were engaged simultaneously. The first set of three, which defined the make up of the Metrics Team, were a cognitive perspective, an operational perspective, and a technical perspective. We also established five "threads" of functional capability to study the technologies: structured discovery, link and group understanding, decision-making with corporate memory, collaborative problem solving, and context-aware visualization. Finally, we established models of the operational context (mission, goals, and resources) for each of the collaboration groups, called "scenarios". The metrics for these analyses were derived using an explicit model based on a UML metrics meta-model. Presenting aggregated analyses in a way that fosters understanding without losing the appreciation of the complexities of the program is itself a complex task. Stones have the ability to organize the anecdotes surrounding events of interest in ways that the goals, values, and rules of the observed behaviors are revealed clearly. Our attempt at building such a "story" used a concept map as a framework.