As car hire and sharing service is popular in transportation market, secure driver identity verification is attracting more attention. However, the current verification mechanism focuses on performing authentication operations in cloud server before drivers get access to the car, which results in potential privacy security issues. In this paper we present a privacy-aware architecture, MobiDIV, which is a client-only scheme where all sensitive data is processed locally on the driver’s smartphone. To achieve real-time and robust driver identification during the driving life cycle, an efficient face features extractor is proposed in MobiDIV. Specifically, 2 three-stream neural network using the proposed efficient SqueezeNet structure are trained on our synthesized dataset for different in-car uncertainties (pose, motion blur, non-alignment and low illumination). During authentication, only an adaptable embedding model is selected and conducted on phone for continuous feature extraction. The anomaly operation monitoring algorithm is then applied to optical signal generated by phone flash for secure identity reidentification and verification failure message transmission. This allows us to further ensure the privacy of driver facial images without compromising on the real-time identity verification. We perform extensive experiments on various datasets. Compared with most SOTA deep neural networks on real-world open datasets, we achieve similar verification accuracy with fewer parameters and floating-point calculations. On the challenging synthetic test datasets, we even achieve a higher average verification accuracy. To assess the MobiDIV in-depth, the proposed model is integrated in car-sharing platform ICICV-E100 and the obtained results show the feasibility of our system.