Low-lying baryon masses using $N_f=2$ twisted mass clover-improved fermions directly at the physical pion mass