PIIM: Method of Identifying Malicious Workers in the MapReduce System with an Open Environment

MapReduce is widely utilized as a typical computation model of mass data processing. When a MapReduce framework is deployed in an open computation environment, the trustworthiness of the participant workers becomes an important issue because of security threats and the motivation of subjective cheating. Current integrity protection mechanisms are based on replication techniques and use redundant computation to process the same task. However, these solutions require a large amount of computation resource and lack scalability. A probe injection-based identification of malicious worker (PIIM) method is explored in this study. The method randomly injects the probes, whose results are previously known, into the input data and detects malicious workers by analyzing the processed results of the probes. A method of obtaining the set of workers involved in the computation of each probe is proposed by analyzing the shuffle phase in the MapReduce programming model. An EnginTrust-based reputation mechanism that employs information on probe execution is then designed to evaluate the trustworthiness of all the workers and detect the malicious ones. The proposed method operates at the application level and requires no modification to the MapReduce framework. Simulation experiments indicate that the proposed method is effective in detecting malicious workers in large-scale computations. In a system with 100 workers wherein 20 of them are malicious, a detection rate of above 97% can be achieved with only 500 randomly injected probes.