Storage Mining: Where IT Management Meets Big Data Analytics

The emerging paradigm shift to cloud based data center infrastructures imposes remarkable challenges to IT management operations, e.g., due to virtualization techniques and more stringent requirements for cost and efficiency. On one hand, the voluminous data generated by daily IT operations such as logs and performance measurements contain abundant information and insights which can be leveraged to assist the IT management. On the other hand, traditional IT management solutions cannot consume and exploit the rich information contained in the data due to the daunting volume, velocity, variety, as well as the lack of scalable data mining and machine learning frameworks to extract insights from such raw data. In this paper, we present our on-going research thrust of designing novel IT management solutions by leveraging big data analytics frameworks. As an example, we introduce our project of Storage Mining, which exploits big data analytics techniques to facilitate storage cloud management. The challenges are discussed and our proof-of-concept big data analytics framework is presented.