Beyond quadratic error: Case-study of a multiple criteria approach to the performance assessment of numerical forecasts of solar irradiance in the tropics

Abstract As the penetration of photovoltaic power in the electrical grid increases, reliable forecasts of solar irradiance are becoming ever more critical. For forecasting horizons beyond a few hours, Numerical Weather Prediction models (NWP) are stapled to yield the best results. For potential users of NWP forecasts, choosing which model to use can be challenging, as a large palette of NWP models and configurations is available. To tackle this issue, several studies have conducted systematic comparisons between available models and configurations. Most of these studies, however, only evaluate forecast skills with a small number of metrics, such as the Root Mean Square Error (RMSE) or the Mean Average Error (MAE). To better understand the nature and specificities of various NWP models, we developed an evaluation approach that combines multiple-criteria analysis, Fourier analysis and classification of daily profiles of irradiance. In this paper, we present this novel approach and demonstrate its strengths by applying it to a practical use-case, using 108 different NWP forecasts models based on different parametrizations of the mesoscale NWP Weather Research and Forecasting model (WRF) in Singapore. Our approach allowed a clearer overview of the forecasts’ skills, as well as better discrimination between models. It also significantly improved our understanding of the nature of the NWP errors. In particular, WRF forecasts in Singapore were found unfit to timely resolve irradiance at an hourly scale, but better adapted to predict daily profiles of irradiance. Furthermore, the proposed multiple criteria approach was applied to a sensitivity analysis of WRF physical schemes, providing a better insight into the impact of each model. This work shows that it is critical to go beyond RMSE when evaluating forecasts and that more holistic approaches such as the one proposed here should be preferred.

[1]  David Pozo-Vázquez,et al.  Macroscopic cloud properties in the WRF NWP model: An assessment using sky camera and ceilometer data , 2015 .

[2]  Vivian Lee,et al.  The Canadian Global Environmental Multiscale model on the Yin‐Yang grid system , 2011 .

[3]  W. Briggs.  Statistical Methods in the Atmospheric Sciences , 2007 .

[4]  John S. Kain,et al.  The Kain–Fritsch Convective Parameterization: An Update , 2004 .

[5]  G. Grell,et al.  A North American Hourly Assimilation and Model Forecast Cycle: The Rapid Refresh , 2016 .

[6]  William C. Skamarock,et al.  A time-split nonhydrostatic atmospheric model for weather research and forecasting applications , 2008, J. Comput. Phys..

[7]  H. Glahn,et al.  The Use of Model Output Statistics (MOS) in Objective Weather Forecasting , 1972 .

[8]  Andreas Kazantzidis,et al.  Evaluation of WRF shortwave radiation parameterizations in predicting Global Horizontal Irradiance in Greece , 2016 .

[9]  J. Dudhia.  Numerical Study of Convection Observed during the Winter Monsoon Experiment Using a Mesoscale Two-Dimensional Model , 1989 .

[10]  H. Niino,et al.  An Improved Mellor–Yamada Level-3 Model: Its Numerical Stability and Application to a Regional Prediction of Advection Fog , 2006 .

[11]  B. Stram.  Key challenges to expanding renewable energy , 2016 .

[12]  Dazhi Yang,et al.  Day-Ahead Solar Irradiance Forecasting in a Tropical Environment , 2015 .

[13]  Z. Janjic.  The Step-Mountain Eta Coordinate Model: Further Developments of the Convection, Viscous Sublayer, and Turbulence Closure Schemes , 1994 .

[14]  S. Sen,et al.  Opportunities, barriers and issues with renewable energy development – A discussion , 2017 .

[15]  M. Thatcher,et al.  Assessing the value of simulated regional weather variability in solar forecasting using numerical weather prediction , 2017 .

[16]  P. Ineichen,et al.  A new airmass independent formulation for the Linke turbidity coefficient , 2002 .

[17]  W. Collins,et al.  Radiative forcing by long‐lived greenhouse gases: Calculations with the AER radiative transfer models , 2008 .

[18]  Zavisa Janjic,et al.  Scientific documentation of the NCEP nonhydrostatic multiscale model on the B grid (NMMB). Part 1 Dynamics , 2012 .

[19]  T. Hoff,et al.  Validation of short and medium term operational solar radiation forecasts in the US , 2010 .

[20]  Andreas Kazantzidis,et al.  Performance evaluation of radiation sensors for the solar energy sector , 2016 .

[21]  Syafrudin Masri,et al.  Challenges of integrating renewable energy sources to smart grids: A review , 2015 .

[22]  Joanne Simpson,et al.  An Ice-Water Saturation Adjustment , 1989 .

[23]  P. Welch.  The use of fast Fourier transform for the estimation of power spectra: A method based on time averaging over short, modified periodograms , 1967 .

[24]  David W. Scott,et al.  Multivariate Density Estimation and Visualization , 2012 .

[25]  H. Niino,et al.  Development of an Improved Turbulence Closure Model for the Atmospheric Boundary Layer , 2009 .

[26]  Hadrien Verbois,et al.  Solar irradiance forecasting in the tropics using numerical weather prediction and statistical learning , 2018 .

[27]  Stephen J. Lord,et al.  Evaluation of the NCEP Global Forecast System at the ARM SGP Site , 2006 .

[28]  J. A. Ruiz-Arias,et al.  Comparison of numerical weather prediction solar irradiance forecasts in the US, Canada and Europe , 2013 .

[29]  Francisco J. Santos-Alamillos,et al.  Evaluation of the WRF model solar irradiance forecasts in Andalusia (southern Spain) , 2012 .

[30]  H. Pedro,et al.  Benefits of solar forecasting for energy imbalance markets , 2016 .

[31]  G. Thompson,et al.  A Study of Aerosol Impacts on Clouds and Precipitation Development in a Large Winter Cyclone , 2014 .

[32]  S. E. Haupt,et al.  WRF-Solar: Description and Clear-Sky Assessment of an Augmented NWP Model for Solar Power Prediction , 2016 .

[33]  David Pozo-Vázquez,et al.  Assessing the Surface Solar Radiation Budget in the WRF Model: A Spatiotemporal Analysis of the Bias and Its Causes , 2016 .

[34]  Thomas Reindl,et al.  Energy meteorology for accurate forecasting of PV power output on different time horizons , 2017 .

[35]  K. Taylor.  Summarizing multiple aspects of model performance in a single diagram , 2001 .

[36]  J. Dudhia,et al.  A Fast All-sky Radiation Model for Solar applications (FARMS): Algorithm and performance evaluation , 2016 .

[37]  M. Diagne,et al.  Review of solar irradiance forecasting methods and a proposition for small-scale insular grids , 2013 .

[38]  Hadrien Verbois,et al.  Probabilistic forecasting of day-ahead solar irradiance using quantile gradient boosting , 2018, Solar Energy.

[39]  S. Freitas,et al.  A scale and aerosol aware stochastic convective parameterization for weather and air quality modeling , 2013 .

[40]  J. Kleissl,et al.  Evaluation of numerical weather prediction for intra-day solar forecasting in the continental United States , 2011 .

[41]  Hans-Georg Beyer,et al.  Irradiance Forecasting for the Power Prediction of Grid-Connected Photovoltaic Systems , 2009, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[42]  Mathieu David,et al.  Probabilistic Solar Forecasting Using Quantile Regression Models , 2017 .

[43]  S. E. Haupt,et al.  The Role of Unresolved Clouds on Short-Range Global Horizontal Irradiance Predictability , 2016 .

[44]  Bob Glahn,et al.  MOS Uncertainty Estimates in an Ensemble Framework , 2009 .

[45]  P. Blanc,et al.  Towards a standardized procedure to assess solar forecast accuracy: A new ramp and time alignment metric , 2017 .

[46]  Charles Elkan,et al.  Using the Triangle Inequality to Accelerate k-Means , 2003, ICML.

[47]  I. Kioutsioukis,et al.  High resolution WRF ensemble forecasting for irrigation: Multi-variable evaluation , 2016 .

[48]  B. Hodge,et al.  The value of day-ahead solar power forecasting improvement , 2016 .

[49]  Bangyin Liu,et al.  Online 24-h solar power forecasting based on weather type classification using artificial neural network , 2011 .

[50]  B. Silverman.  Density estimation for statistics and data analysis , 1986 .

[51]  J. Kindle,et al.  Summary diagrams for coupled hydrodynamic-ecosystem model skill assessment , 2009 .

[52]  M. Borga,et al.  Space-time variability of climate variables and intermittent renewable electricity production – A review , 2017 .

[53]  R. Piwko,et al.  Look Before You Leap: The Role of Energy Storage in the Grid , 2012, IEEE Power and Energy Magazine.

[54]  J. A. Ruiz-Arias,et al.  Benchmarking of different approaches to forecast solar irradiance , 2009 .

[55]  R. Inman,et al.  Solar forecasting methods for renewable energy integration , 2013 .

[56]  A. Meij,et al.  GHI calculation sensitivity on microphysics, land- and cumulus parameterization in WRF over the Reunion Island , 2018 .

[57]  M. Chin,et al.  Sources and distributions of dust aerosols simulated with the GOCART model , 2001 .

[58]  C. Fountoukis,et al.  Simulating global horizontal irradiance in the Arabian Peninsula: Sensitivity to explicit treatment of aerosols , 2018 .

[59]  Max J. Suarez,et al.  A Solar Radiation Parameterization for Atmospheric Studies , 2013 .

[60]  Flore Remouit,et al.  Variability assessment and forecasting of renewables: A review for solar, wind, wave and tidal resources , 2015 .