Building an information system from the Web

To allow users to truly use the Web to construct personal information systems, users must be able to write their own applications to retrieve, massage, combine, and store information from Web servers. Information providers cannot know all the ways their information can be used; that is determined by the collectivity of users. If users cannot write their own applications, then Web access will remain a tedious and manual process. After describing two small applications we show that the Web architecture, based on HTML, a display-oriented language for describing pictures, does not support client applications very well; the structure and marking of a page does not describe its information in a way easily understood by software. Nevertheless, because the information is mostly textual and was designed to convey that information to a human, it is often possible to retrieve needed information from a page. We describe our implementation, written in Scheme, which queries pages using set predicates, extracts information, and uses that to query further Web pages. Extensions of this approach can combine this information with the clients other local resources. Finally, the same tools are applicable to more sophisticated markup systems, arch as SGML or its Web-oriented offspring XML