The ensemble of surrogate models is increasingly implemented in practice for its more flexibility and robustness compared to the individual surrogate models. In this work, a novel pointwise ensemble of surrogate models named the optimization-based two-layer pointwise ensemble of surrogate model (OTL-PEM) is proposed. In the OTL-PEM, the framework of two-layer surrogate models is defined, where the data-surrogate models containing different types of individual surrogate models are to fit the given dataset, while the weight-surrogate models are modeled based on the cross-validation errors aiming to fit the pointwise weights of different individual surrogate models. To avoid the negative influence of the poor individual surrogate models, the model selection problem is transformed into several optimization problems which can be solved easily by the mature optimization algorithm to eliminate the globally poor surrogate models. In addition, the optimization space is extracted to alleviating the predictive instability caused by the extrapolation of the weight-surrogate models. Forty test functions are used to select the appropriate hyperparameters of the OTL-PEM, and to evaluate the performance of the OTL-PEM. The results indicate that the OTL-PEM can provide more accurate and robust approximation performance compared with individual surrogate models as well as other ensembles of surrogate models.