We demonstrate that, through coherent measurement of the transmitted terahertz frequency electric fields, broadband (0.3 - 8 THz) time-domain spectroscopy can be used to measure far-infrared vibrational modes of a range of drugs-of-abuse and high explosives that are of interest to the forensic and security services. Our results indicate that absorption features in these materials are highly sensitive to the structural and spatial arrangement of the molecules. Terahertz frequency spectra are also compared with high-resolution low-frequency Raman spectra to assist in understanding the low-frequency inter- and intra-molecular vibrational modes of the molecules.