SMEMO: Social Memory for Trajectory Forecasting