We report the measurement of thermal conductance of silicon nanowires at low temperature. It is demonstrated that the roughness at the nanometer scale plays a crucial role for the phonon transport in low-dimensional samples. To this end, using e-beam lithography, nanowires of size 200 nm by 100 nm and 10 microm long have been nanofabricated. Their thermal properties have been measured using the 3 omega method between 0.3 and 6 K. The change in the temperature behavior of the thermal conductance (quadratic temperature dependence of K(T)) is a signature of an intermediate regime lying between the classical Casimir regime and the quantum regime. The Casimir-Ziman model is used to show that this specific behavior originates in mesoscopic samples where the dominant phonon wavelength becomes commensurate to the characteristic length of the roughness of the nanowire surfaces.