Detecting qualitatively different behaviours in complex models

We suggest that one of the main purposes of modelling is to explore the potential dynamical behaviours a system can display. Within this view, we aim to discriminate model behaviours which appear to be qualitatively different given a problem at hand. This approach fits nicely within a pre-cautionary approach to ecological and social problems aimed to inform policy-makers on the range of scenarios a policy may need to address. Numerical modelling is increasingly being used to inform policy-making with examples including resource management, biodiversity conservation, global warming mitigation and economic policy. The interpretation of modelling results thus has the potential to profoundly affect our environment and millions of people. Currently, there is an on-going discussion among modelling practitioners on what a model output represents, how it should be interpreted and what its overall scientific significance is: views cover a continuum between two extremes: one suggests that models can provide only a qualitative understanding of the modelled process and their output simply offers insight into general trends; another sees a model as a virtual laboratory in which real processes are roughly mimicked and whose outcome can be interpreted as predictions. Somewhere in between these views, a number of practitioners suggest that the purpose of modelling is to explore the potential behaviours a system can display. This is the framework we adopt in this work and we try to design an algorithm able to discriminate different model behaviours from a numerical model output. There are three main challenges in implementing this approach: the first one is how to define and discriminate different behaviours. This is clearly problem-specific and depends not only on the purpose of the analysis but also on the kind of output a model produces. We define a number of simple measures able to detect both local and global features in the model output and we discuss how the method could be extended to qualitative model output, that is a subjective evaluation of the model output performed by an expert user. The second challenge is how to detect different behaviours, which we address via a search in a high- dimensional input space. Finally, once a set of different behaviours have been found, these need to be presented to the user and if many of such behaviours have been detected some sort of classification and simplification is also needed. We employ a Self-Organised Map to allow an approximate visual representation of our results. While these tools do not provide for an 'exact' analysis of the results, we believe they allow a potential decision-maker to obtain a rough picture of the variability and the range of behaviours policies may need to address. In general, the proposed approach should not be seen as an avenue to obtain firm problem-independent answers on a model behaviour, rather as a tool to highlight difference in model behaviours and provide their rough categorization. This information can then be used to guide a more focussed search of the model space aimed at answering problem-dependent specific questions in more details. Here we provide a proof of concept on a number of numerical models and discuss an extension to participatory modeling.