Latency Matters: Real-Time Action Forecasting Transformer