Automated detection of anomalous shipping manifests to identify illicit trade

We describe an approach to analyzing anomalies in trade data based on the identification of cluster outliers. The approach uses unsupervised machine learning methods to discover semantically coherent clusters of shipping records in large collections of trade data. Trade data with cluster annotations are then used as input to a supervised machine learning algorithm to train and evaluate a classification model capable of identifying members of each cluster. The evaluation of this classification model provides an assessment of cluster coherence. Outliers are identified for each cluster by measuring the Euclidean distance from each member of the cluster to the cluster centroid, and then selecting a percentile threshold to identify shipping records with extreme distances from the cluster centroid. We describe a specific application of this approach to a dataset of 2.36M records for containerized shipments, with specific reference to the detection of anomalies potentially related to nuclear smuggling. Results show that this approach succeeds in finding semantically coherent clusters of shipping records, and identifying outliers that may help facilitate the detection of illicit trade.