Big Brother is Watching You—But in a Good Way

In any modern desktop environment the glyph compositor-where raw text information is combined with font information and other attributes to render rasterized component images-is part of the software´s core functionality. In this paper we present work that shows it is computationally feasible to apply full-text indexing in real-time to the live stream of glyph compositor operations generated by a user´s interaction with their desktop environment. By embedding indexing functionality at such a level, we effectively get to “see” (and more importantly remember) all the text that is drawn on the user´s screen. With elements reminiscent of the Memex, we illustrate the technique in use through a personal digital library we have developed that enriches (through text-searching and context) the user´s desktop experience by letting them go back in time to view information that had previously been displayed. We achieved this by augmenting our dynamically updated text index with time-stamped snapshots of the desktop. By recording the (x, y) positions of the text at the time it is rendered, the snapshots have a semi-live feel, whereby text can be selected for copy-and-paste operations for further use. Moreover, windows-even if they were hidden behind others at the time the text was rendered-can be brought to the front and their text accessed.