Are redshift-space distortions actually a probe of growth of structure?