SL(2, ℤ) action on QFTs with ℤ2 symmetry and the Brown-Kervaire invariants