Hidden non-Markovian reward Models : Virtual stochastic sensors for hybrid systems

We are interested in partially observable hybrid systems whose discrete behavior is stochastic and unobservable, and for which samples of some of the continuous variables are available. Based on these samples of the continuous variables, we show how the hidden discrete behavior may be reconstructed computationally, which was previously not possible. The paper shows how Hidden non-Markovian Models (HnMM) can be augmented with arbitrary rate and impulse rewards to model these partially observable hybrid systems. An HnMM analysis method is adapted to find the probability of a sample sequence for a given model, as well as likely system behaviors that caused the observation. Experiments illustrate the analysis method and the possible complexity of the reward measure through a medical example and one from computer gaming. The paper extends the class of partially observable systems analyzable via virtual stochastic sensors into the continuous realm for the first time.