Performance Evaluation of Various Storage Formats for Clinical Data Repositories

Fast access to clinical data is necessary when performing real-time predictions of medical events. A clinical data repository (CDR) therefore requires an efficient format for storing data so it can meet the access demands of prediction algorithms for clinical decision support. We have developed a new hybrid entity-attribute-value (EAV) storage format for CDRs that is compared with the common simple and multidata-type EAV formats. Their performance is evaluated with regard to querying millions of data points from clinical sources to assess the amount of storage space the data use, the speed with which the data can be obtained, and the complexity of the queries required to retrieve the data. Performance results showed that our hybrid EAV approach provides a nice balance of the simple and multidata-type formats that scales well as the amount of data in the CDR increases. The hybrid format is faster, uses less complex queries, and allows the data to remain in its original data type while using the same amount of storage space as the simple and multidata-type formats.