Mechanization of operations within a noncommutative algebraic structure: Application to the unitary transformation of the Hamiltonian of a polyatomic molecule