Physical realization of a quantum spin liquid based on a complex frustration mechanism