DocumentCode :
3585014
Title :
Deep Order Statistic Networks
Author :
Rennie, Steven J. ; Goel, Vaibhava ; Thomas, Samuel
fYear :
2014
Firstpage :
124
Lastpage :
128
Abstract :
Recently, Maxout networks have demonstrated state-of-the-art performance on several machine learning tasks, which has fueled aggressive research on Maxout networks and generalizations thereof. In this work, we propose the utilization of order statistics as a generalization of the max non-linearity. A particularly general example of an order-statistic non-linearity is the “sortout” non-linearity, which outputs all input activations, but in sorted order. Such Order-statistic networks (OSNs), in contrast with other recently proposed generalizations of Maxout networks, leave the determination of the interpolation weights on the activations to the network, and remain conditionally linear given the input, and so are well suited for powerful model aggregation techniques such as dropout, drop connect, and annealed dropout. Experimental results demonstrate that the use of order statistics rather than Maxout networks can lead to substantial improvements in the word error rate (WER) performance of automatic speech recognition systems.
Keywords :
multilayer perceptrons; speech recognition; statistical analysis; OSN; WER performance improvement; annealed dropout; automatic speech recognition systems; deep-order statistic networks; drop connect; dropout; input activations; interpolation weights; linear network; max-nonlinearity; maxout networks; model aggregation techniques; network activation; order-statistic nonlinearity; sortout nonlinearity; word error rate performance improvement; Acoustics; Annealing; Conferences; Detectors; Neural networks; Speech; Training; Deep Neural Networks; Maxout Networks; Multi-Layer Perceptrons; Order Statistic Networks; Rectified Linear Units;
fLanguage :
English
Publisher :
ieee
Conference_Titel :
Spoken Language Technology Workshop (SLT), 2014 IEEE
Type :
conf
DOI :
10.1109/SLT.2014.7078561
Filename :
7078561
Link To Document :
بازگشت