Joint optimization of multiple optical planes in a stellar coronagraph