A Novel Outlier Detection Algorithm for Distributed Databases

Traditional outlier detection algorithms are designed to apply to centralized databases, not distributed databases. We proposed a novel outlier detection algorithm for distributed databases. Given data assigned to different network nodes of a network platform, where each node has its own memory and hard disc, and the communication between nodes driven by message, the populated data would be non-overlapping. The working way of the network system is a manager-worker mode, that is, that a node as manager is responsible for assigning tasks to worker and querying the results from worker nodes. The algorithm first detected local outliers based on distance on all nodes, and then identified local outliers collected in the central node where a globally screening operation on all local outliers was implemented to achieve really global outliers. To scale the algorithm to massive data and reduce its computing complexity, a data filtering technology was further presented. Experimental results demonstrated that the algorithm effectively and efficiently handled on real and artificial data.