In-beam separation and mass determination of superheavy nuclei. Part I