Quantum key distribution (QKD) has demonstrated a great potential to provide future-proofed security, especially for 5G and beyond communications. As the critical infrastructure for 5G and beyond communications, optical networks can offer a cost-effective solution to QKD deployment utilizing the existing fiber resources. In particular, measurement-device-independent QKD shows its ability to extend the secure distance with the aid of an untrusted relay. Compared to the trusted relay, the untrusted relay has obviously better security, since it does not rely on any assumption on measurement and even allows to be accessed by an eavesdropper. However, it cannot extend QKD to an arbitrary distance like the trusted relay, such that it is expected to be combined with the trusted relay for large-scale QKD deployment. In this work, we study the hybrid trusted/untrusted relay based QKD deployment over optical backbone networks and focus on cost optimization during the deployment phase. A new network architecture of hybrid trusted/untrusted relay based QKD over optical backbone networks is described, where the node structures of the trusted relay and untrusted relay are elaborated. The corresponding network, cost, and security models are formulated. To optimize the deployment cost, an integer linear programming model and a heuristic algorithm are designed. Numerical simulations verify that the cost-optimized design can significantly outperform the benchmark algorithm in terms of deployment cost and security level. Up to 25% cost saving can be achieved by deploying QKD with the hybrid trusted/untrusted relay scheme while keeping much higher security level relative to the conventional point-to-point QKD protocols that are only with the trusted relays.