Chaotic encryption is widely used in the field of image encryption due to its initial value sensitivity, pseudo-randomness, and unpredictability of motion trajectories. In this paper, we propose to attack Lorenz chaotic encrypted system of grayscale images by the deep learning method via Residual Networks which contains skip connection and can automatically analyze the relationships of plaintext and ciphertext. After the training process to a series of input and output plaintext-ciphertext pairs, we can finally recover the image approximate to the plaintext image according to the ciphertext which is independent to the original plaintext-ciphertext pairs set. Numerical simulation verified that the result recovering from the chaotic encrypted system is very good.