A Survey on Risk Forecast Evaluation

Model evaluation is crucial for the …nancial industry and especially for risk management purposes. It provides the means of determining the accuracy of the implemented models while it is a prerequisite for the internal model’s approach. Given that the actual return generating process is unknown, additional statistical assumptions are needed in order to proceed with the model evaluation. These assumptions introduce distortions in the evaluation process, which a¤ect the reliability of results. This paper reviews the major Value at Risk and Expected Shortfall forecast evaluation methods and their performance. While focusing on the regulatory speci…cation, we include various simulation setups to evaluate the e¤ects of model risk and the length of the out-of-sample period. Our …ndings highlight the ine¢ ciencies of the forecast backtesting methods for the regulatory speci…cations, which may eventually lead to the selection of inaccurate and misspeci…ed models. Keywords: Value-at-Risk, Expected Shortfall, Model Accuracy, Backtesting, Forecast Evaluation Acknowledgements: This work was supported by the Economic and Social Research Council [grant number 1494761]. Corresponding author. Ekaterini Panopoulou, Kent Business School, University of Kent, Canterbury CT2 7PE, United Kingdom, Tel.: 0044 1227 824469, Email: A.Panopoulou@kent.ac.uk. 1 Introduction Evaluating/backtesting a risk model consists of ex-post out-of-sample comparisons of actual returns with the model’s forecasted risk measures. It provides the statistical means for determining the accuracy of the plethora of proposed models. Although backtesting is a crucial component of the internal model’s approach, there are no speci…c regulatory recommendations for the type of tests that should be used. On the contrary, the evaluation methodology is freely chosen by the implementing institution. Since the Market Risk Amendment to the Basel I accord in 1996, Value at Risk (VaR) has been the dominant risk measure for market risk quanti…cation. VaR was formally introduced by JP Morgan in 1994 and can be de…ned as the maximum loss for a given signi…cance level q over a speci…c time horizon. In other words, VaR constitutes the q-quantile of the forecasted return distribution. Although the quantile nature of VaR makes it easy to understand and implement, it is also the main source of criticism. Among others, Danielsson et al. (2001) and Dowd (2005) point out two de…ciencies of VaR. First, it represents a simple point on the tails of the distribution and therefore does not provide any information beyond that. Second, VaR is not a coherent risk measure as it is not subadditive for the general case of distributions. These de…ciencies alongside the aftermath of the recent …nancial crisis led to the proposal of the coherent Expected Shortfall (ES ) as a substitute for VaR (BIS (2012)). In theory, ES seems superior and more informative than VaR since it averages the tail losses over the signi…cance level q. However, Embrechts et al. (2014) conclude that both risk measures are estimated statistical quantities, a fact that makes their implementation a subject for future research. With respect to the evaluation of candidate risk models, there is a large body of literature proposing di¤erent approaches. For instance, the results of Diebold et al. (1998) and Berkowitz (2001) established the density evaluation approach, which evaluates the …t of the model’s implied density. Although intuitive, density evaluation requires the disclosure of key information regarding an institution’s returns, thus making it less appealing for the …nancial industry. Within the density evaluation approach, Amisano and Giacomini (2007) advocate towards comparing model’s performance instead of evaluating their …t. The authors argue that, given the absence of a correct proxy for the actual return distribution, evaluating the …t of the model maybe futile. Unlike the aforementioned density evaluation approaches, forecast evaluation tests the model’s accuracy by assessing the properties embedded in the forecasts. This approach was established as the industry benchmark since it provides intuitive motivation, ease of implementation and little demand for sensitive portfolio returns information. Despite its advantages and appeal, the Forecast Evaluation Approach methods su¤er from small sample ine¢ ciencies attributed to the regulatory speci…cations and various methodological assumptions. More in detail, the scarcity of events at the 1% VaR and 1-year out-of-sample period reduce the amount of information input, thus resulting in small power. In addition, the structural and distributional assumptions of the tests reinforce the negative e¤ects on their power. Similarly, model risk emerges as another major source of unreliability since it can distort the results in two possible ways. First, Escanciano and Olmo (2010) and Escanciano and Olmo 1 (2011) suggest that the forecast inherited model risk a¤ects the asymptotic variance of the test statistics. Second, the actual test statistics assumptions may be evaluating a human-designed structure that accounts only for small fragments of the actual return dynamics. Finally, with respect to the risk measure selection there is a large debate regarding the underlying risk measure and its ability of being evaluated. Some academics suggest that ES forecasts can not be evaluated given the measure’s lack of elicitability (see, for example Ziegel (2014)). On the other hand, there is new evidence that elicitability is not necessary for forecast backtesting rather than comparing their performance (see, for example Emmer et al. (2015)). In a recent paper, Nieto and Ruiz (2016) survey the VaR forecasting and backtesting literature without however evaluating the performance of the backtesting methods. Speci…cally, the authors evaluate the performance of various VaR forecasting methods at the 1% coverage level. Their empirical exercise includes di¤erent setups designed to look into the e¤ects of various in-sample and out-of-sample periods. The corresponding results suggest that there is signi…cant variation in the accuracy of the forecasting methods. In addition, the authors conclude that simpler methods with asymmetric volatility dynamics and error distributions are the most competitive. Our work is, in principle, similar to Campbell (2007) and complements Nieto and Ruiz (2016) by focusing on the performance of the backtesting methods. In this paper, we survey the methodological and empirical developments in the risk forecast evaluation approach. Speci…cally, we focus on the regulatory speci…cations of 1% VaR and 2.5% ES forecasts while implementing two di¤erent out-of-sample periods in order to evaluate the small sample properties of the tests. In addition, we utilize various in-sample periods in order to evaluate the e¤ect of forecast estimation risk. Finally, we use the S&P 500 returns to evaluate the forecasts on real …nancial data. Our …ndings suggest that the low power of the tests renders the evaluation results unreliable. Speci…cally, the scarcity of extreme events at the high coverage level distort the size of the tests. The tests are oversized for almost all the cases under scrutiny, thus questioning the validity of their asymptotic distributions. Similarly, the power of the tests is low which again is attributed to the scarcity of events and the forecasting method’s structure. With respect to the latter, the overall results suggest that the reliability of each backtesting method varies with the assumptions of the forecasting method. Finally, model risk reduces even more the power of the methods thus aggravating the mistrust on the backtesting methods. The rest of the paper is structured as follows. In Section 2 we introduce the VaR and ES de…nitions notions and main forecasting approaches. Section 3 describes the evaluation methods for VaR and ES while in Section 4 we perform the small sample evaluation of the tests. Finally, in Section 5 we conduct an empirical implementation on the S&P returns and Section 6 concludes. 2 Risk Forecasting In this section, we describe the risk forecasting procedure and the main forecasting approaches. First, we give a brief description of the …nancial industry’s most common risk measures and