Competing edge structures of Sb and Bi bilayers generated by trivial and nontrivial band topologies