Trace formulas and dynamical zeta functions in the Nielsen theory