In this paper, the combination of particle swarm optimization (PSO) and gravitational search algorithm (GSA) is enhanced by the first-order gradient method and a new optimization algorithm is introduced as GPSG. In metaheuristic methods, some search directions are selected at random and the resulting points gradually progress toward the optimal. Since the gradient direction usually has the largest decrease in the desired function, it is added to the GSA and PSO process to allow for faster and more accurate convergence. By integrating the metaheuristic methods with the gradient directions, a powerful method for optimizing functions has been made possible. Numerous examples of unconstrained problems of mathematical functions of CEC2005 and constrained examples of stress and displacement structural design problems have been chosen to demonstrate the reliability and capability of the presented method.