DocumentCode :
806219
Title :
Enhancing data analysis with noise removal
Author :
Xiong, Hui ; Pandey, Gaurav ; Steinbach, Michael ; Kumar, Vipin
Author_Institution :
Manage. Sci. & Inf. Syst. Dept., Rutgers Univ., Newark, NJ, USA
Volume :
18
Issue :
3
fYear :
2006
fDate :
3/1/2006 12:00:00 AM
Firstpage :
304
Lastpage :
319
Abstract :
Removing objects that are noisy is an important goal of data cleaning as noise hinders most types of data analysis. Most existing data cleaning methods focus on removing noise that is the product of low-level data errors that result from an imperfect data collection process, but data objects that are irrelevant or only weakly relevant can also significantly hinder data analysis. Thus, if the goal is to enhance the data analysis as much as possible, these objects should also be considered as noise, at least with respect to the underlying analysis. Consequently, there is a need for data cleaning techniques that remove both types of noise. Because data sets can contain large amounts of noise, these techniques also need to be able to discard a potentially large fraction of the data. This paper explores four techniques intended for noise removal to enhance data analysis in the presence of high noise levels. Three of these methods are based on traditional outlier detection techniques: distance-based, clustering-based, and an approach based on the local outlier factor (LOF) of an object. The other technique, which is a new method that we are proposing, is a hyperclique-based data cleaner (HCleaner). These techniques are evaluated in terms of their impact on the subsequent data analysis, specifically, clustering and association analysis. Our experimental results show that all of these methods can provide better clustering performance and higher quality association patterns as the amount of noise being removed increases, although HCleaner generally leads to better clustering performance and higher quality associations than the other three methods for binary data.
Keywords :
data analysis; data mining; noise; pattern clustering; HCleaner; clustering-based technique; data analysis; data mining; distance-based technique; hyperclique-based data cleaner; local outlier factor; noise removal; Cleaning; Computer Society; Computer errors; Data analysis; Error analysis; Noise level; Object detection; Proteins; Text analysis; Index Terms- Data cleaning; hyperclique pattern discovery; local outlier factor (LOF); noise removal.; very noisy data;
fLanguage :
English
Journal_Title :
Knowledge and Data Engineering, IEEE Transactions on
Publisher :
ieee
ISSN :
1041-4347
Type :
jour
DOI :
10.1109/TKDE.2006.46
Filename :
1583581
Link To Document :
بازگشت