According to analyses of the real-time performance of the visual prosthesis image processing system, down-sampling is a key factor which influences the processing speed. Based on these analyses, the algorithm of grid sampled averaging was proposed for down-sampling. Then the effect of grid parameters on the result and algorithm complexity was evaluated by using the regional-averaging algorithm as a reference. Our research shows that the grid-averaging algorithm could reduce the computation burden by 90% or more, with no significant difference from the regional-averaging algorithm.