Gradient-Free Neural Network Training via Synaptic-Level Reinforcement Learning

An ongoing challenge in neural information processing is: how do neurons adjust their connectivity to improve task performance over time (i.e., actualize learning)? It is widely believed that there is a consistent, synaptic-level learning mechanism in specific brain regions, such as the basal ganglia, that actualizes learning. However, the exact nature of this mechanism remains unclear. Here we investigate the use of universal synaptic-level algorithms in training connectionist models. Specifically, we propose an algorithm based on reinforcement learning (RL) to generate and apply a simple biologically-inspired synaptic-level learning policy for multi-layer perceptron (MLP) models. In this algorithm, the action space for each MLP synapse consists of a small increase, decrease, or null action on the synapse weight, and the state for each synapse consists of the last two actions and global reward signals. A binary reward signal indicates either an increase or decrease in the model loss between the previous two iterations. This algorithm yields a static synaptic learning policy that enables simultaneous training of over 20,000 parameters (i.e. synapses) and consistent MLP convergence when applied to simulated decision boundary matching and optical character recognition tasks. The static policy is robust as it produces faster and more consistent training relative to the adaptive policy and is agnostic to activation function, network shape, and task. The trained networks yield character recognition performance comparable to identically shaped networks trained with gradient descent. 0 hidden unit character recognition tests yielded an average validation accuracy of 88.28%, 1.86±0.47% higher than the same MLP trained with gradient descent. 32 hidden unit character recognition tests yielded an average validation accuracy of 88.45%, 1.11±0.79% lower than the same MLP trained with gradient descent. The approach has two significant advantages in comparison to traditional gradient descent-based optimization methods. First, the robustness of our novel method and lack of reliance on gradient computations opens the door to new techniques for training difficult-to-differentiate artificial neural networks such as spiking neural networks (SNNs) and recurrent neural networks (RNNs). Second, the method’s simplicity provides a unique opportunity for further development of local rule-driven multi-agent connectionist models for machine intelligence analogous to cellular automata. Preprint. Under review. ar X iv :2 10 5. 14 38 3v 1 [ cs .N E ] 2 9 M ay 2 02 1