Big Data in Memory: Benchmarking in Memory Database Using the Distributed Key-Value Store for Constructing a Large Scale Information Infrastructure

The Universal Communication Research Institute (UCRI), NICT conducts research and development on universal communication technologies: multi-lingual machine translation, spoken dialogue, information analysis and ultra-realistic interaction technologies, through which people can truly interconnect, anytime, anywhere, about any topic, and by any method, transcending the boundaries of language, culture, ability and distance. To enhance the universal communication technology, we are trying to develop a large-scale information infrastructure which collects and stores diverse information including huge volumes of web pages from the networks. The one of most important key technologies to realize a large-scaled information infrastructure is a distributed in memory database system. In this paper, we introduced a large-scale information infrastructure, mainly explaining a distributed in-memory database system "okuyama" which is a key technology on our project. We examined the I/O performance of the in memory storage, which verified whether if "okuyama" meet requirements for the infrastructure. Furthermore we give a blueprint of cluster systems on which the infrastructure will be constructed.