Tighter Lower Bounds for Shuffling SGD: Random Permutations and Beyond