Improving data quality: eliminating dupes & I-D-ing those spurious links

Dirty data arise as a result of abbreviations, data entry mistakes, duplicate records, missing fields and so forth. This problem is aggravated when multiple data sources need to be integrated. Data cleaning refers to a series of processes employed to deal with detecting and removing errors and inconsistencies from data. Given the "garbage in, garbage out" principle, clean data is crucial for database integration, data warehousing and data mining. Data cleaning is a necessary step prior to the knowledge discovery process. We have reviewed a knowledge-based framework that provides for the definition of duplicate identification rules. We have described a context-based approach to identify these spurious links in the data.