Automatic Generation of Ontology from the Deep Web

The term "deep Web" refers to Web pages that are not accessible to search engines, e.g., because those Web pages are dynamically generated in response to queries through Web forms or Web services. The existing automated Web crawlers cannot index these pages, thus they are hidden from the Web search engines. Our goal is to properly annotate such deep Web services (i.e. content generation interfaces of hidden Web sources) with semantic indexing by constructing domain-specific ontologies to represent the contents of the deep Web sources. The fully automatic derivation of ontologies from Web sources without human review is to date a challenging research issue. We present a novel approach to automatically building a large, yet domain-specific, ontology by interweaving sub-taxonomies of WordNet with domain-specific information extracted from deep Web service pages. Our algorithms extract domain concepts from deep Web sources which are augmented with concepts and relationships from WordNet to construct ontology fragments. Structurally, these are directed acyclic graphs (DAGs). An iterative process of extracting WordNet concepts and relationships and bridging concept gaps is used to tie together disparate domain concepts and ontology fragments into one ontology. Using eight domains (airfares, jobs, etc.) from a well-known test-bed, our algorithms constructed an ontology of 1692 concepts from deep Web sources and 4434 concepts from WordNet. This ontology is expressed in the OWL format to support semantic Web searches.