Annotating network trace data for anomaly detection research

Anomaly detection holds significant promise for automating network operations and security monitoring. Many detection techniques have been proposed. To evaluate and compare such techniques requires up to date datasets, useful truth data and the ability to record the outputs of the techniques in a common format. Existing datasets for network anomaly detection are either limited / aged or lacking in truth data. This paper presents a new annotation format allowing network datasets to be annotated with arbitrary event data. Use of the new format is demonstrated in a method to create new datasets that retain more information than a simple network capture. The supporting tools for the annotation format allow for incorporating events from multiple different sources. The ability to record and share network data and detected anomalies is a key component in moving anomaly detection research forward.