Calculating Necessary Neuron Gains for Winner-Take-All Networks

T. X Brown

NASA Tech Briefs, NPO-18640 Aug. 1991

Abstract:
The winner-take-all (WTA) and its generalizations arise repeatedly in
neural circuitry. We address the question of how large a gain is
necessary for a given WTA circuit to guarantee its WTA functionality.
In general, the gain required is an increasing function of the network
size. So conversely, we answer the question of given neurons with a
specific gain, how large a circuit can we build. In hardware
implementations, the gain is limited to finite values.  These questions
are then important when designing massively parallel WTA realizations.
This paper shows that the required gain increases as a superlinearly
function of the number of neurons, with solutions derived for various
common neuron transfer functions. These results provide a basis for
comparing possible neuron designs. Of the functions considered, the
hyperbolic tangent increases the slowest with an $O(N\log(N))$
dependency. This greatly reduces the required gain; dramatically
increasing the achievable circuit sizes over other common transfer
functions.