In order to adequately preserve the spatial information of panchromatic (PAN) image and the spectral information of multispectral (MS) image, this study proposes a method to learn the spectral information of MS based on the PAN image. In the spectral learning model, the PAN is degraded to the spatial resolution of the MS. Then, a deep convolutional neural network (CNN) is adopted to learn the spectral information between the degraded PAN and the original MS. In addition, the spectral evaluation index—spectral angle mapper (SAM) is used for controlling the spectral losses. Finally, the high spatial resolution MS (HMS) can be obtained through the trained model by using the original PAN as the test input. Besides, seven representative pan-sharpening algorithms and seven widely recognized objective fusion metrics are used to compare and evaluate the performance on the WorldView-2 experimental data, respectively. The results show that the proposed method achieves the purpose of pan-sharpening well, especially maintaining the optimal spectral information.