Cohomological support loci of varieties of Albanese fiber dimension one