Modular bootstrap for D4-D2-D0 indices on compact Calabi-Yau threefolds