SNAP: Towards a Validation of the Social Network Assembly Pipeline

A key problem for social network analysis is the lack of ground-truth data upon which to validate an analysis. Consider for example community-finding algorithms. The ``communities´´ identified by such algorithms are typically justified on the basis of their structural properties, rather than on their ability to recover communities which can be independently verified. A ground truth of actual community data isn´t always available and at best only partial ground-truth community information is. However, this problem isn´t unique to community-finding algorithms. In previous publications, we introduced an automated Social Network Assembly Pipeline we refer to as SNAP. This is intended for the large scale actor identification, tie interference and strength measurement of social networks from non-relational data sets. In this paper we describe a validation study of SNAP through an intensive user-study of a portion of the individuals in the network. Individuals are asked to validate the network relationships uncovered by SNAP and where misclassified relationships are found, the individuals are interviewed in order to determine the underlying cause of the misclassification. The findings provide feedback on the rules through which relationships are inferred. For instance, it becomes clear that an error in actor identification can result in a propagation of this error though the network relations leading to follow-on relationship misclassifications. Also, we observe how outliers lead to a propagation of error in the inferred network. The results help us validate and invalidate different hypotheses we have about SNAP and suggests domain specific rule-sets for SNAP.