Flexible development and deployment of large scale data processing in MAGIC