Aiming at the problems of slow recognition, low efficiency and degree of automation in handwritten letter recognition system at present, a handwritten letter recognition system based on extreme learning machine is designed in this paper. The system is implemented by mixed programming with MATLAB and visual studio, it can reads, normalize, binarize and extract the handwritten letter images. The real-time interactive recognition of handwritten letters can be realized on the basis of training the simple pictures by using the identification model of the extreme learning machine algorithm. The experimental results show that the handwriting recognition system based on extreme learning machine designed in this paper can recognize 98.82% of handwritten letters and greatly reduce learning and testing time. Compared with BP neural network and other recognition algorithms, its training times have been reduced by hundreds or even thousands of times. At the same time, there is no manual intervention in the entire learning and testing process, which improves the automation of handwriting recognition.