Optimizing the echo state network with a binary particle swarm optimization algorithm

The echo state network (ESN) is a novel and powerful method for the temporal processing of recurrent neural networks. It has tremendous potential for solving a variety of problems, especially real-valued, time-series modeling tasks. However, its complicated topologies and random reservoirs are difficult to implement in practice. For instance, the reservoir must be large enough to capture all data features given that the reservoir is generated randomly. To reduce network complexity and to improve generalization ability, we present a novel optimized ESN (O-ESN) based on binary particle swarm optimization (BPSO). Because the optimization of output weights connection structures is a feature selection problem and PSO has been used as a promising method for feature selection problems, BPSO is employed to determine the optimal connection structures for output weights in the O-ESN. First, we establish and train an ESN with sufficient internal units using training data. The connection structure of output weights, i.e., connection or disconnection, is then optimized through BPSO with validation data. Finally, the performance of the O-ESN is evaluated through test data. This performance is demonstrated in three different types of problems, namely, a system identification and two time-series benchmark tasks. Results show that the O-ESN outperforms the classical feature selection method, least angle regression (LAR) method in that its architecture is simpler than that of LAR.