Internet of Things (IoT) is a key enabler of Industry 4.0 with networked devices providing sensor data to help manage, automate, streamline and optimize assets, operations and processes. In such industrial IoT settings, reliability and process experts spend a considerable amount of time in creating accurate ground-truth data to assist with the inferencing capabilities of Artificial Intelligence (AI) engines. This process can be time-consuming and sometimes inaccurate, depending on the complexity of data. Accurate expert annotated data is the foundation for many AI applications because data needs to be classified on several bases, for instance into ‘normal’, ‘abnormal’ or ‘pre-abnormal’ states. Such problem formulations can be appropriately addressed using Active Learning (AL) techniques. We propose an AL framework capable of handling two practical challenges: oracle uncertainty and quantification of model performance in the absence of ground truth. Consequently, the proposed approach addresses uncertainties within AL techniques by fusing information pertaining to expertise levels of the human annotators and their confidence levels corresponding to the annotation provided.