Abstract Feature selection is an important procedure that is used in data mining to extract valuable information from large quantities of data. Existing penalization methods use a single penalty function to select important features. However, these methods do not yield sufficiently accurate predictions and selection outcomes. Therefore, construction of a concise and efficient prediction model would be beneficial. In this study, we propose a novel penalty function using a ridge and minimax concave penalty to overcome the limitations of individual penalty functions. Furthermore, we introduce a robust penalized feature selection method with Huber loss function, which is implemented by a local approximation algorithm. The theoretical properties of the algorithm have been described. Simulated and real-world data analyses are used to demonstrate the efficacy of the proposed method.