Condensing Multilingual Knowledge with Lightweight Language-Specific Modules