Efficient Approximation of Optimal Control for Markov Games – early arXiv version
Markus N. Rabe, Sven Schewe, and Lijun Zhang
The success of probabilistic model checking for discrete-time Markov decision processes and continuous-time Markov chains has led to rich academic and industrial applications. The analysis of their combination in continuous-time Markov decisions processes, however, is currently restricted to toy examples. This is due to the fact that current analysis techniques for time-bounded reachability require a running time linear in the required precision. For the high precision usually sought (for example, six to ten digits), this simply renders these techniques infeasible. We discuss a surprising combination of discretisation and partial unravelling, which leads to memoryful near optimal schedulers that can be computed in time linear only in the square or cube root of the required precision. The proposed techniques also reduce the dependency on the expected number of discrete transitions within the given time bound significantly. Our techniques naturally extend to the analysis of continuous-time Markov games.