The generalized optical memory effect