Bistable probabilities: a unified framework for studying rationality and irrationality in classical and quantum games