Graph network for learning bi-directional physics