Trust Your IMU: Consequences of Ignoring the IMU Drift