Reversible data hiding (RDH) is a research hotspot nowadays. By RDH, after data extraction, the cover image can be restored without information loss. Among numerous existing RDH techniques, multiple histograms modification (MHM) is a general reversible embedding framework, and it is experimentally verified better than the traditional single histogram based methods. However, the expansion-bins-determination process for MHM is conducted through naive exhaustive search, which is time consuming. Based on this consideration, a fast expansion-bins-determination method for MHM is proposed in this paper. Specifically, to determine the optimal expansion bins, instead of solving the optimization problem of discrete variables, we consider a general form of this problem with differentiable objective function and real variables, so that advanced analysis tools such as Lagrange multiplier can be utilized. By the proposed approach, compared with the original MHM, the expansion bins can be determined quickly with only a tiny performance loss, and thus the practicality of MHM is improved.