Lifetime of Double Occupancies in the Fermi-Hubbard Model