A parallel lexical analyzer for multi-core machines

Lexical analysis is the first and foremost step in the compilation process. The basic aim of this step is to convert stream of characters / symbols into words called tokens. These tokens are used by the other phases of a compiler. Many tools have been developed in the past that generate the tokenizer automatically. These tools are best suited for sequential processing. With the advent of multi-core processors it is possible to parallelize tokenization by exploiting the parallel constructs of the languages. In this paper we propose a parallel lexical analyzer which exploits loops and processor affinity concept to achieve parallelism in lexical analysis on multi-core machines. The results show substantial improvement in lexical analysis phase as compared to sequential version.