We describe the optimization of a complex video encoder system based on target architecture. We implemented the MPEG-4 encoder using hardware/software codesign approach, mapped together based on a target architecture. We proposed a target architecture template and an optimization methodology. In our design flow, we searched for a bottleneck module constraining the system. After investigating the computational complexity, quality, and the simplicity of algorithms, we chose the best algorithm for hardware implementation, and then mapped the selected algorithm onto the hardware with different architecture, what does the best architecture for the algorithm and which is the best architecture of components. We chose one of the architectures meet the constraints and also made tradeoffs among speed, chip area, and memory bandwidth for different architecture. The proposed system architecture was used to reduce the design decisions and iterations, provided flexible and scalable systems. The evaluations resulted in effective optimization of the motion estimation module and better tradeoffs that optimized the overall system.