Orchestration of Distributed Storage Targets through Storage Flows

Distributed data storage is a topic of growing importance due to the mounting pressure to find the right balance between capacity, cost, privacy and other non-functional properties. Compared to central storage on physical media, on the network or in a cloud storage service, advanced data distribution techniques offer additional safety, security and performance. On the downside, these advantages come with a much higher complexity regarding the choice and configuration of where to store which parts of the data, and subsequent verification of where which data had been stored. Often, the storage targets must be configured individually while a centrally and locally accessible configuration interface with an appropriate propagation and verification mechanism would be more suitable. The complexity is further increased by additional data pre-processing tasks which are selectively applied to some of the targets. Compression, encryption and deduplication are typically present in pre-processing. With Storage Flows, we propose a new concept to manage distributed storage flows through systematic orchestration. The flows connect clients flexibly with intermediate data pre-processing tasks and finally the storage targets. We show that Storage Flows can be formalised and demonstrate their practical usefulness with implemented configuration and verification tools.