Improving the Discovery and Characterization of Hidden Variables by Regularizing the LO-net

This paper presents an extension of the regularized neural network architecture, called the LO-net. LO-net is a neural network architecture that can infer both the existence and values of hidden variables in streaming multivariate time series. The core idea is to initially make predictions with one network (the observable or O-net) based on a time delay embedding, following this with a gradual reduction in the temporal scope of the embedding that forces a second network (the latent or L-net) to learn to approximate the value of a single hidden variable, which is then input to the O-net based on the original time delay embedding. The latent network sometimes learns to approximate the predicted target output from the original network in the LO-net architecture. To prevent this situation, a penalty term is introduced that is added to the error for the latent network. The penalty term is formulated to penalize the latent network when it learns the target output of the original network. A distance penalty term has been shown to improve the prediction performance over the unregularized network. This paper introduces a new penalty, called the decor relation penalty, which proves to be better for domains with periodic data.