Diffraction analysis of three-dimensional volume gratings with arbitrary boundaries