Heart sounds are an important basis for evaluating heart disease. In recent years, auxiliary diagnosis technology of cardiovascular disease based on the detection of heart sound signals has become a research hotspot. A lightweight automatic heart sound classification method is presented in this study. It only needs to perform simple preprocessing on the heart sound data, and a neural network model is constructed to automatically extract the time–frequency features of heart sound data: where the convolution module extracts the frequency-domain characteristics, the loop module extracts the time-domain characteristics, and the features are then fused in series and parallel. Finally, the classification and recognition of heart sounds are realized based on the fusion features. At the same time, methods, such as group convolution, global average pooling, and gated loop mechanisms, are used to reduce the parameters and training time of the neural network. The experimental data from the PhysioNet database are used for testing. The recognition accuracy of the series feature fusion heart sound classification model is 95.00%, and the model size is 0.6 MB. In addition, the parallel feature fusion heart sound classification model is 95.50%, and the model size is 1.36 MB. Therefore, the lightweight heart sound classification model designed in this study contains fewer parameters and higher recognition accuracy. This makes it suitable for deployment in embedded devices. It also has important research significance for the development of portable heart sound detection equipment.