Improving procedures for evaluation of connectionist context-free language predictors

This letter shows how seemingly minor differences in training and evaluation procedures used in recent studies of recurrent neural networks as context free language predictors can lead to significant differences in apparent network performance. We suggest standard evaluation procedures whose use would facilitate better reproductibility and comparability.