Revealing magnetic ground state of a layered cathode material by muon spin relaxation and neutron scattering experiments