We report a de Haas-van Alphen (dHvA) study of the electronic structure of Al doped crystals of MgB$_2$. We have measured crystals with $\sim 7.5$% Al which have a $T_c$ of 33.6 K, ($\sim 14$% lower than pure MgB$_2$). dHvA frequencies for the $\sigma$ tube orbits in the doped samples are lower than in pure MgB$_2$, implying a $16\pm2%$ reduction in the number of holes in this sheet of Fermi surface. The mass of the quasiparticles on the larger $\sigma$ orbit is lighter than the pure case indicating a reduction in electron-phonon coupling constant $\lambda$. These observations are compared with band structure calculations, and found to be in excellent agreement.