Higher Lower Bounds for Near-Neighbor and Further Rich Problems

We convert cell-probe lower bounds for polynomial space into stronger lower bound for near-linear space. Our technique applies to any lower bound proved through the richness method. For example, it applies to partial match, and to near-neighbor problems, either for randomized exact search, or for deterministic approximate search (which are thought to exhibit the curse of dimensionality). These problems are motivated by search in large data bases, so near-linear space is the most relevant regime. Typically, richness has been used to imply Omega(d/lg n) lower bounds for polynomial-space data structures, where d is the number of bits of a query. This is the highest lower bound provable through the classic reduction to communication complexity. However, for space n lg ^O(1)n, we now obtain bounds of Omega(d/ lg d). This is a significant improvement for natural values of d, such as lg^O(1) n. In the most important case of d = Theta (lg n), we have the first superconstant lower bound. From a complexity-theoretic perspective, our lower bounds are the highest known for any static data-structure problem, significantly improving on previous records