Towards unification of lepton and quark mass matrices from double covering of modular $A_4$ flavor symmetry