Convergence of Multi-Scale Reinforcement Q-Learning Algorithms for Mean Field Game and Control Problems