Dementia is a disease characterized by cognitive impairment that leads to incoherent or illogical thoughts and speech. There are attempts to identify dementia through speech analyses, but there is a dearth of research on casual conversation analysis. This work examined communication impairment detection of people with early-stage memory loss, including mild dementia and mild cognitive impairment. The data sets included semi-structured interviews from two studies conducted at the University of Washington (UW), the DementiaBank’s Pitt Corpus, and the ADReSS Challenge at INTERSPEECH 2020. We applied Transformer-based deep learning models to automatically extract linguistic features for identifying individuals with dementia. Our results showed the models’ abilities on detecting linguistic deficits with the best mean F1-score of 76% on the Pitt Corpus, 84% on the ADReSS, 90% on the augmented ADReSS, and 74% on the UW transcripts. The results suggest the potential possibility of a more flexible examination setting, casual semi-structured individual or group interview, for detecting incoherent or illogical thoughts and speech in patients with dementia.