Quantum projection filter for a highly nonlinear model in cavity QED