A Systematic Study on Matrix Models for Chern-Simons-matter Theories