Hidden invariance of last passage percolation and directed polymers