Evaluation of simple statistical criteria to qualify a simulation

Statistical and deterministic simulation modelling rely on a complex process made of trials, errors, and gradual improvement of the simulations. The major problem is to be able to quantify the quality of the simulations in order to know if a modification of the concepts, the laws simulating the processes, or the parameters improve it. To try to quantify the quality of simulations using a mathematical criterion we focus on simple linear regression parameters: the values of the slope (a) and the y-intercept (b). The estimated values of these parameters differ depending on which kind of regression model (model I or II) is used. An artificial dataset illustrates that ordinary least-squares regression (OLS; model I regression) leads to results that are not those expected; but using major axis regression (MA; model II regression) instead of OLS leads to the correct answer. The value of a, when it significantly differs from 1, indicates a difference between observed and simulated values proportional to the values of the variable. The value of b, when it significantly differs from 0, indicates a systematic and constant difference between observations and simulations. Taking into account the values of a and b, we define four possible outcomes which allow, at first, to define the quality of a simulation without considering the coefficient of determination, r2: (i) a n.s.d. (not significantly different from) 1 and b n.s.d. 0 (perfect agreement between observations and simulations), (ii) a n.s.d. 1 and b s.d. 0 (significant constant difference between observations and simulations) or a s.d. 1 and a s.d. 0 and b n.s.d. 0 (differences proportional to the values of the variable), (iii) a s.d. 1 and a s.d. 0 and b s.d. 0 (superimposition of a constant difference and a proportional difference), and (iv) a n.s.d. 0 (no relation between simulations and observations). The value of r2 is used to rank two simulations pertaining to the same group. That classification of the quality of the simulations is applied to a real-data example: a simulation of the temporal change in chlorophyll a in a high-rate algal pond.