The Hirzebruch-Mumford volume for the orthogonal group and applications