Speeding up Large-Scale Financial Recomputation with Memoization

Quantitative financial analysis requires repeated computations of the same functions with the same arguments when prototyping trading strategies; many of these functions involve resource intensive operations on large matrices. Reducing the number of repeated computations either within a program or across runs of the same program would allow analysts to build and debug trading strategies more quickly. We built a disk memoization library that caches function computations to files to avoid recomputation. Anymemoization solution should be easy to use, minimizing the need for users to think about whether caching is appropriate, while at the same time giving them control over speed, accuracy, and space used for caching. Guo and Engler proposed a similar tool that does automatic memoization by modifying the python interpreter, while the packages Jug and Joblib are distributed computing tools that have memoization options. Our library attempts to maintain the ease of use of the above packages for memoization, but at the same time give a higher degree of control of how caching is done for users who need it. We provide the same basic features as these other libraries, but allow control of how hashing is done, space usage for individual functions and all memoization, refreshing memoization for a specific function, and accuracy checking. This should lead to both increased productivity and speed increases for recomputation. We show that for several financial calculations, including Markowitz Optimization, Fama French, and the Singular Value Decomposition, memoization greatly speeds up recomputation, often by over 99%. We also show that by using xxhash, a non-cryptographic hash function, instead of md5, and avoiding equality checks, our package greatly outperforms joblib, the best current package.