Sensitivity of Neutron Star Observations to Three-nucleon Forces