Probing the quantum phase transition in Mott insulator BaCoS2 tuned by pressure and Ni substitution