Memory-Based High-Performance Optimization for High Concurrent Data-Intensive Problems

Large-scale data-intensive problems characterized with high task concurrency have overwhelmingly been on the rise, calling for higher computing efficiency. One solution for this is to make use of high-performance computers. In this paper, we propose an optimization by constructing distributed file system in big memory of nodes and rearranging compute nodes to greatly reduce repeated and redundant I/O and enhance memory usage as well as task parallelism. An example of information identification in a large database is given to illustrate its running process. A mathematical analysis is also presented to prove better performance gain of the proposed solution.