Miniaturized image classifiers are potential for revolutionizing their applications in optical communication, autonomous vehicles, and healthcare. With deep diffractive neuron networks trained subwavelength structures, we demonstrate image recognitions by a passive silicon photonic metasystem. The metasystem implements high-throughput vector-by-matrix multiplications, enabled by 103 passive subwavelength phase shifters as weight elements in 1 mm2 footprint. The large weight matrix size incorporates the fabrication variation related uncertainties, and thus the pre-trained metasystem can perform machine learning tasks without post-tuning. A 15-pixel spatial pattern classifier reaches near 90% accuracy with femtosecond inputs. The metasystem’s superior parallelism (1015 bit/s) dramatically expand data processing capability of photonic integrated circuits, towards next generation low latency and low power photonic accelerators compatible with complementary metal-oxide-semiconductor manufacturing.