Molecular beam study of surface roughening transition