Using density functional theory (DFT) and density functional perturbation theory (DFPT), we investigate the stability and response functions of CsH$_2$PO$_4$, a ferroelectric material at low temperature. This material cannot be described properly by the usual (semi-)local approximations within DFT. The long-range e$^-$-e$^-$ correlation needs to be properly taken into account, using, for instance, Grimme's DFT-D methods, as investigated in this work. We find that DFT-D3(BJ) performs the best for the members of the dihydrogenated alkali phosphate family (KH$_2$PO$_4$, RbH$_2$PO$_4$, CsH$_2$PO$_4$), leading to experimental lattice parameters reproduced with an average deviation of 0.5 %. With these DFT-D methods, the structural, dielectric, vibrational and mechanical properties of CsH$_2$PO$_4$ are globally in excellent agreement with the available experiments ($<$ 2% MAPE for Raman-active phonons). Our study suggests the possible existence of a new low-temperature phase for CsH$_2$PO$_4$, not yet reported experimentally. Finally, we report the implementation of DFT-D contributions to elastic constants within DFPT.