Towards the development of numerical procedure for control of connected Markov chains

The system of controlled time-inhomogeneous Markov chains (MCs) is considered. The principal problem related to this kind of systems and could be called as the "curse of dimension" appears as a necessity of solving a system of ordinary differential equations of high dimension. Moreover, even the software development for such systems is a serious issue since these equations are linked and the standard parallelization approaches in existing software packages are not very effective. Meanwhile, we noticed that the minimization procedure needed for the right-hand side (RHS) of this system may be easily parallelized by independent minimization in each equation. As an example we consider the management of linked dams under non-stationary seasonally changing random inflows/outflows and customers´ demands. The current state of each dam is described by the state of continuous-time MC corresponding to the water level. So the state of the dams system is represented in tensor form. The connection of MCs is a result of the controlled flow between dams. The aim of the control is to maintain the required water levels on the weather conditions and to satisfy the customers´ demands. The general approach is based on the solution of Bellman type equation in tensor form. This equation may be reduced to the system of ordinary differential equations. We suggest here the automatic procedure for the generation of this system and also the approach to the minimization of the RHS which may be realized for each state of MC independently.