Bi-level variable selection in semiparametric transformation models with right-censored data

In this article, we investigate bi-level variable selection approaches in semiparametric transformation models when a grouping structure of covariates is available. This large class of transformation models includes the Cox proportional hazards model and proportional odds model as special cases. For this class of models, there are only a few works on variable selection and all the selection methods are at individual variable level. To fill the gap of variable selection at both group and individual levels, we propose a penalized nonparametric maximum likelihood estimation method with three different penalties, i.e., group bridge (GB), adaptive group bridge (AGB) and composite group bridge (CGB), and develop their respective computational algorithms. Further, we prove that the resulting estimators from AGB and CGB have desirable oracle properties. Our simulation studies demonstrate that all the three penalties work well in bi-level variable selection, while AGB and CGB outperform GB when within-group sparsity is present. The proposed methods are applied to two real datasets for illustration.