Efficient Continual Learning with Modular Networks and Task-Driven Priors