Fishtail effect and the vortex phase diagram of high-entropy alloy superconductor