Attending to Remember and Remembering to Attend