Training threshold neural networks by extreme learning machine and adaptive stochastic resonance

Abstract

Threshold neural networks are highly useful in engineering applications due to their ease of hardware implementation and low computational complexity. However, such threshold networks have non-differentiable activation functions and therefore cannot be trained by standard gradient-based algorithms. To circumvent this limitation, here we propose a hybrid training algorithm for threshold neural networks. The proposed hybrid training algorithm has two distinguishing features: the structural transformation of the hidden layer enables threshold networks to benefit from a noise-boosted learning capability via adaptive stochastic resonance (ASR), and by using the fast learning algorithm of the extreme learning machine (ELM) suitable generalization performance ensues for the threshold networks. Experimental results on regression and on multiclass classification demonstrate the realizability and practical efficiency of the proposed hybrid training algorithm, thereby demonstrating the beneficial role of artificial noise injection in threshold neural networks.