Topological characterization of higher-order topological insulators with nested band inversion surfaces