The optical constants of heavily doped silicon are determined between 1.5 and 4.1 eV with use of polarization modulation ellipsometry. For photon energies less than 3.4 eV it is found that there is a substantial increase in the optical absorption coefficient for As-doped silicon, but a much smaller increase for B-doped or P-doped silicon. It is believed that this is due to d-electron admixture into the conduction band when the dopant is As.