Observational Clues to Brown Dwarf Origins