Phase space optimization of quantum representations: Three-body systems and the bound states of HCO