Exotic $U(1)$ symmetries, duality, and fractons in 3+1-dimensional quantum field theory