Forecasting Political Conict in Asia and the Middle East using Latent Dirichlet Allocation Models

Latent Dirichlet allocation models are a relatively new computational classification algorithm. In its standard application to document classification, the model assumes each document to be composed of a mixture of multiple, overlapping topics each with a typical set of words, and classification is done by associating words in a document with the latent topics most likely to have generated the observed distribution of those words. I apply this technique to the problem of political forecasting by assuming that the stream of events observed between a dyad of actors is a mixture of a variety of different political strategies and standard operating procedures (for example escalation of repressive measures against a minority group while simultaneously making efforts to co-opt the elites of that group). By identifying the dominant strategies being pursued at time t, one gets information that can be used to forecast likely patterns of interaction at a later time t + k. This approach is applied to event data generated for 29 Asian countries in the Integrated Conflict Early Warning System project for 1998-2010 to forecast the ICEWS conflict measures for rebellion, insurgency, ethno-religious violence, domestic political conflict and international conflict at a six month lead time, and to the Israel-Palestine and Israel-Lebanon dyads from the KEDS Levant data set for 1979-2009. In random samples balancing the occurrence of negative and positive outcomes on the dependent variable, LDA combined with a logistic model predicts with around 60% to 70% accuracy in in-sample evaluation in both data sets, and improves very substantially on the sensitivity of the classification compared with simple logistic models in full samples. A supervised version of LDA, however, does not provide much improvement over the unsupervised version, and shows some pathological behaviors. Some structure can be found in the factors, though more work is needed on this.