Automated Discovery of Unknown Unknowns

The "unknown unknowns" are factors affecting a decision that are not known to exist at the beginning of the evaluation of a situation. While humans (analysts, children, etc.) seem to have this reasoning capability, the automation of this process is more difficult. In fact, until now it has been an unresolved research problem. Using automated reasoning and learning algorithms (separately and in combination), we leveraged extremely rich structures from a knowledge base. To do this, we took differences between what was specified as known in the knowledge base, and what new evidence suggested. These differences, dubbed "unknown unknowns", were of three types: new hypotheses that might explain a situation, new links that explain previously unknown relationships between facts, and new story fragments that describe the significance of previously unidentified collections of factors that might drive a decision. The basic approach enabled us to prototype various different methods for the automated discovery of unknown unknowns (ADUU), depending on the characteristics of the decisions. We successfully demonstrated these methods in three domains: strategic planning and analysis, information operations (contractually), and anti-terrorism