A Black-box Approach for Non-stationary Multi-agent Reinforcement Learning