Traders’ behavioral coupling and market phase transition