DocumentCode :
3721415
Title :
Knowledge reduction method based on information entropy for port big data using MapReduce
Author :
Weiping Cui; Lei Huang
Author_Institution :
School of Economics and Management, Beijing Jiaotong University, China
fYear :
2015
fDate :
7/1/2015 12:00:00 AM
Firstpage :
1
Lastpage :
6
Abstract :
With the volume of port data growing at an unprecedented rate, analyzing and extracting knowledge from large-scale data sets have become a new challenge in decision making. But, the application of standard data mining tools in such data sets is not straightforward. Hence, we develop a parallel large-scale knowledge reduction method based on rough set for knowledge acquisition using MapReduce in this paper. It designs and implements the Map and Reduce functions using data and task parallelism. Then, it constructs the parallel algorithm framework model for knowledge reduction using MapReduce, which can be used to compute a reduct for the algorithms based on information entropy. The experimental results demonstrate that the proposed parallel knowledge reduction method can efficiently process massive datasets on Hadoop platform, with highly speed up the classification process and largely reduce the storage requirements.
Keywords :
"Ports (Computers)","Information entropy","Data mining","Big data","Knowledge acquisition","File systems","Algorithm design and analysis"
Publisher :
ieee
Conference_Titel :
Logistics, Informatics and Service Sciences (LISS), 2015 International Conference on
Type :
conf
DOI :
10.1109/LISS.2015.7369695
Filename :
7369695
Link To Document :
بازگشت