When Is Affirmative Action Fair?: On Grievous Harms and Public Remedies