Text-based systems and information management: artificial intelligence confronts matters of scale

Many of the more ambitious goals of artificial intelligence have proved unattainable because of the failure of the many small, successful systems to scale up. The general use of technologies such as natural language interfaces and expert systems has done little to alleviate the basic difficulties and overwhelming cost of knowledge engineering. At the same time, emerging text processing techniques, including data extraction from text and new text retrieval methods, offer a means of accessing stores of information many times larger than any organized knowledge base or database. Although knowledge acquisition from text is at the heart of the information management problem, interpreting text, paradoxically, requires large amounts of knowledge, mainly about the way words are used in context. In other words, before intelligent text processing systems can be trained to mine for useful knowledge, they must already have enough knowledge to interpret what they read. The point at which there is “enough”, is still a matter of debate, as no real program seems close to having enough knowledge to achieve general human-like understanding. Current research in large-scale natural language processing has come, rightly, to focus on lexical acquisition as the key to future progress. Unfortunately, the current state of the art is quite far from the recipe for acquiring knowledge about words, because it leans too heavily on resources that are available, without consideration for what is needed