Topological phase transition in layered magnetic compound MnSb2Te4: Spin-orbit coupling and interlayer coupling dependeces