Spatially Varying Nanophotonic Neural Networks