CEFLS: A Cost-Effective File Lookup Service in a Distributed Metadata File System

As large file systems increasingly grow in size, metadata operations become one of the major performance bottlenecks that constrain the overall I/O performance. Previous analysis on I/O workloads shows the file lookup operation makes up a large proportion of metadata operations. Existing optimizations for lookup operations such as MHS method employ the directory lookup table (DLT) to avoid directory traversal. However, the inefficient design of DLT produces large amount of storage cost and rename overhead, not suitable for large file systems. In this paper, we present a cost-effective file lookup service (CEFLS) for a distributed metadata file system. Our method benefits from efficient partition method and structures to increase the cache efficiency for DLT. Extensive simulations show that the percentages of cached directories with CELFS can be increased by factors of up to 305 and 279 percent compared with MHS when the cache size on each metadata server is configured as 1GB and 2GB, respectively. Meanwhile, CELFS can also significantly reduce the average latency for both file lookup and directory rename operations.