An optimal randomised cell probe lower bound for approximate nearest neighbour searching

We consider the approximate nearest neighbour search problem on the Hamming cube {0, 1 }^d. We show that a randomised cell probe algorithm that uses polynomial storage and word size d^O(1) requires a worst case query time of Ω (log log d/ log log log d). The approximation factor may be as loose as 2^{log 1 - η}d for any fixed η > 0. This generalises an earlier result (Chakrabarti et al., 1999) on the deterministic complexity of the same problem and, more importantly, fills a major gap in the study of this problem since all earlier lower bounds either did not allow randomisation according to Chakrabarti et al. (1999) and Liu (2003) or did not allow approximation according to Borodin et al. (1999), Barkol and Rabani (2000), and Jayram et al. (2003). We also give a cell probe algorithm which proves that our lower bound is optimal. Our proof uses a lower bound on the round complexity of the related communication problem. We show, additionally, that considerations of bit complexity alone cannot prove any nontrivial cell probe lower bound for the problem. This shows that the richness technique (Miltersen et al., 1995) used in a lot of research around this problem would not have helped here. Our proof is based on information theoretic techniques for communication complexity, a theme that has been prominent in research by Chakrabarti et al. (2001), Bar-Yossef et al. (2002), Sen (2003) and Jain et al. (2003). In particular, we make heavy use of the round elimination and message compression ideas in the work of Sen (2003) and Jain et al. (2003), and also introduce a technique which we call message switching.