Multimodal speaker identification in legislative discourse

A first-of-its-kind platform, Digital Democracy1 offers a searchable archive of all statements made in US state legislative hearings in four American states (California, New York, Texas and Florida) covering one third of the US population. The purpose of the platform is to increase government transparency in state legislatures. It allows citizens to follow state lawmakers, lobbyists, and advocates as they debate, craft, and vote on policy proposals. State hearings in the U.S. are typically recorded on video and broadcast on cable TV stations, but they are not transcribed or indexed. No official written records of these exist anywhere. Digital Democracy creates professional quality transcripts and provides them publicly in a searchable and video-synchronized format. In this paper, we focus on one of the main challenges in video transcription: identifying the speaker. While transcription provides searchable content of state hearings, people can only follow the dynamics of a debate if they know who said what and when during a hearing. Our speaker identification approach applies well-known voice recognition, deep-learning-based face recognition, and text understanding approaches to an ever growing pool of legislative hearing videos. All speaker recognition predictions from voice, face, and text are combined by a ranking scheme, and ultimately integrated back into the transcription process. We describe the current architecture, technologies, and multimodal fusion used for speaker identification within the Digital Democracy project. We show that our multimodal fusion approach increases "top 5" legislator identification accuracy to 80% in California and 58% in New York.