Hysteretic magnetoresistance and unconventional anomalous Hall effect in the frustrated magnet TmB4