Magnetic properties of an anomalous phase transition around 5 K in YbSb