On the Fine-Grained Complexity of Rainbow Coloring