One of the promising systems to realize quantum computation is a hybrid system where a superconducting flux qubit plays the role of a quantum processor and the nitrogen-vacancy- $({\mathrm{NV}}^{\ensuremath{-}})$ center ensemble is used as a quantum memory. We have theoretically and experimentally studied the effect of magnetic fields on this hybrid system, and found that the lifetime of the vacuum Rabi oscillation is improved by applying a few mT magnetic field to the ${\mathrm{NV}}^{\ensuremath{-}}$ center ensemble. Here, we construct a theoretical model to reproduce the vacuum Rabi oscillations with and without magnetic fields applied to the ${\mathrm{NV}}^{\ensuremath{-}}$ centers, and we determine the reason why magnetic fields can affect the coherent properties of the ${\mathrm{NV}}^{\ensuremath{-}}$ center ensemble. From our theoretical analysis, we quantitatively show that the magnetic fields actually suppress the inhomogeneous broadening from the strain in the ${\mathrm{NV}}^{\ensuremath{-}}$ centers.