Reducing memory latency using a small software driven array cache

From the programming viewpoint, data references can be classified into two types: array reference and non-array references. Array references have relatively strong spatial locality while non-array references have relatively strong temporal locality. However, in current data cache designs, the hardware cannot distinguish between these two types of references. Both types of data are stored in the same cache space and all cache control mechanisms such as prefetching are applied to array references as well as to non-array references. As a result, data cache performance is often not satisfactory. The large working set of array references with weak temporal locality interferes with the small working set of non-array references with strong temporal locality and replaces them away from cache. Applying hardware driven data prefetching scheme to array references might improve cache performance. However, when the same scheme is applied to non-array references, cache performance might be lost due to serious cache pollution. To solve all these problems, this paper proposes a new software driven cache design, called the array cache. The main idea is to use a separate cache space to store and handle array references with constant strides that are prefetched accurately with the help of the compiler and with extremely low runtime overhead