Phase transition between the quantum spin Hall and insulator phases in 3D: emergence of a topological gapless phase