Passive Operator Networks