Affinity-Aware Graph Networks