Measurements with a TRISTAN prototype detector system at the “Troitsk nu-mass” experiment in integral and differential mode