Differential cryptanalysis of a novel image encryption algorithm based on chaos and Line map

In this paper, we analyze the security of a chaos-based image encryption algorithm. We find that the algorithm is vulnerable to differential cryptanalysis. The differential cryptanalysis demonstrates that the security of the original scheme depends only on the permutation key instead of on all of the keys, which makes the key space of the cryptosystem greatly reduced. Specifically, $$(H\times W+1)$$(H×W+1) chosen plain-images can reveal the equivalent permutation key for one-round encryption, where $$H\times W$$H×W is the size of the binary image. The two-way differential comparison method is proposed to break two-round cryptosystem. Basing on differential cryptanalysis, we propose a codebook attack under chosen-ciphertext conditions, design the codebook with $$(H\times W)$$(H×W) differential binary images, and totally break multi-round cryptosystem using XOR operation of $$O(H\times W)$$O(H×W) images. The simulation results indicate that the designed codebook attack is effective. Furthermore, we summarize some characteristics of a class of permutation–diffusion systems that also are vulnerable to the differential attack and the codebook attack.