The brown dwarf atmosphere monitoring (BAM) project - I. The largest near-IR monitoring survey of L and T dwarfs