Inferring Actions and Observations from Interactions