Surface states and topological invariants in three-dimensional topological insulators: Application to Bi 1 − x Sb x