Contribution analysis: a technique for assigning responsibilities to hidden units in connectionist networks

Contributions, the products of hidden unit activations and weights, are presented as a viable tool for investigating the inner workings of neural nets. Using a scaled-down version of NETtalk, a fully automated method for summarizing in a compact form both local and distributed hidden-unit responsibilities is demonstrated. Contributions are shown to be more useful for ascertaining hidden-unit responsibilities than either weights or hidden-unit activations. Among the results yielded by contribution analysis: for the example net, redundant output units are handled by identical patterns of hidden units, and the amount of responsibility a hidden unit takes on is inversely proportional to the number of hidden units.