Adaptive Control and Cost Sensitivity of Stochastic Systems with Continuous State and Parameters, and Logical Controls

We consider event-driven stochastic control problems, in which the state {Xt} takes values in a continuum, and the control actions {Ut} are selected from a finite (event) set. Furthermore, lack of knowledge about certain quantities in the model, e.g. transition probability matrices, is accounted for via a parameterization by a quantity ¿, which also takes values in a continuum. More specifically, the model for the stochastic systems that we consider is that of a discrete-time controlled Markov process (CMP), parameterized by the quantity ¿. This model is described by the quadruplet ≪X, U, ¿, P≫, where the state space X and parameter set ¿ are complete and separable metric spaces, the event (or control) set U is a finite set, and P(· | x, u; ¿) is the transition (stochastic) kernel that gives the distribution of Xt+1, when Xt = x, Ut = u, and the (unknown) value for the parameter is ¿; see [4], [6], [9], for details on CMP. The characteristic of special interest in our model is that while the state and unknown parameters take values in a continuum, the control actions are drawn from a finite set, i.e., these are logical or Boolean quantities. This situation arises in many problems of interest.