Unidirectional protein recruitment is essential for viral latency