Lattice instability and competing spin structures in the double perovskite insulator Sr2FeOsO6.