NLM’s System Description for the Fourth i2b2/VA Challenge

The NLM team participated in the concept extraction, assertion classification and relation extraction tasks of the Fourth i2b2 Challenge. After exploring rulebased approaches to the tasks, the team has resorted to supervised machine learning methods. The results varied by task. The concept extraction task suffered from over-training, achieving 0.60 0.74 F1-score in the evaluation (compared to 0.93 cross-validation results on the training set). The performance of the assertion classifiers in the evaluation was consistent with that in training, achieving an overall 0.93 F1score. An SVM classifier demonstrated mixed behavior in the relation extraction task: it was overtrained for two relations with few positive examples, and the results were consistent with training for the remaining relations (achieving an overall 0.67 F1score on the test set.) The improvements in relation extraction due to feature selection observed in crossvalidation evaluations during training were not observed in the test evaluation. The assertion and relation classifiers applied to concepts extracted by our system maintained their respective performance, but the scores were much lower than for the groundtruth based evaluation.