Limit set of inhomogeneous Ornstein-Uhlenbeck processes, destabilization and annealing