A new model of continuous-time Markov processes and impulse stochastic control

We propose a composite model for Markov processes. The state space of a composite Markov process consists of two parts, J and J¿. When the process is in J¿, it evolves like a continuous-time Levy process; and once the process enters J, it makes a jump instantly according to a transition function like a direct-time Markov chain. The composite Markov process provides a new model for impulse stochastic control problem, with the instant jumps in J modeling the impulse control feature (e.g., selling or buying stocks in the portfolio management problem). With this model, a new approach may be developed to the impulse stochastic control problem. The approach is based on a direct comparison of the performance of any two policies, and hence the results are intuitive clear. The new approach also provides some new insights leading to new research topics in the area, such as sample-path-based policy iteration and gradient based optimization.