Generalized hypertree decompositions: NP-hardness and tractable variants