Fermi's golden rule beyond the Zeno regime