Search for Gravitational Waves from the Coalescence of Subsolar Mass and Eccentric Compact Binaries