Instability of the Liouville property for quasi-isometric graphs and manifolds of polynomial volume growth