Numerical methods for the optimization of nonlinear stochastic delay systems, and an application to internet regulation

The Markov chain approximation method is an effective and widely used approach for computing optimal values and controls for stochastic systems. It was extended to nonlinear (and possibly reflected) diffusions with delays in a recent book. The path, control, and reflection terms can all be delayed. In models of communications systems, reflection terms correspond to buffer overflows, and to the adjustments needed to keep the buffers and rates nonnegative and bounded. The book concentrated on convergence proofs for general algorithms of promising forms. If the control and/or reflection terms are delayed, the memory requirements can make the problem intractable, since these have little or no regularity. For such models, the problem was recast in terms of a “wave” or “transportation” equation, to get practical algorithms with very much reduced computational needs. The overall approach to getting usable algorithms is outlined. The methods are applied to an ideal model of a communications system with transportation delay. and it is seen that nearly optimal controls for appropriately chosen criteria can improve the performance considerably and be robust in the face of changing operating conditions. The data illustrate the potential usefulness of numerical methods for nonlinear stochastic systems with delays. It opens up new avenues of control for consideration.