Replay in minds and machines

7 Experience-related brain activity patterns have been found to reactivate during sleep, wakeful 8 rest, and brief pauses from active behavior. In parallel, machine learning research has found that 9 experience replay can lead to substantial performance improvements in artificial agents. Together, 10 these lines of research have significantly expanded our understanding of the potential computa11 tional benefits replay may provide to biological and artificial agents alike. We provide an overview 12 of findings in replay research from neuroscience and machine learning and summarize the compu13 tational benefits an agent can gain from replay that cannot be achieved through direct interactions 14 with the world itself. These benefits include faster learning and data efficiency, less forgetting, 15 prioritizing important experiences, as well as improved planning and generalization. In addition 16 to the benefits of replay for improving an agent’s decision-making policy, we highlight the less-well 17 studied aspect of replay in representation learning, wherein replay could provide a mechanism to 18 learn the structure and relevant aspects of the environment. Thus, replay might help the agent to 19 build task-appropriate state representations. 20