Direct measurement of optical quasidistribution functions: Multimode theory and homodyne tests of Bell's inequalities