Quantum effects on the dynamics of a two-mode atom-molecule Bose-Einstein condensate