A Graph Clustering Approach to Localization for Adaptive Covariance Tuning in Data Assimilation Based on State-Observation Mapping

An original graph clustering approach for the efficient localization of error covariances is proposed within an ensemble-variational data assimilation framework. Here, the localization term is very generic and refers to the idea of breaking up a global assimilation into subproblems. This unsupervised localization technique based on a linearized state-observation measure is general and does not rely on any prior information such as relevant spatial scales, empirical cutoff radii or homogeneity assumptions. Localization is performed via graph theory, a branch of mathematics emerging as a powerful approach to capturing complex and highly interconnected Earth and environmental systems in computational geosciences. The novel approach automatically segregates the state and observation variables in an optimal number of clusters, and it is more amenable to scalable data assimilation. The application of this method does not require underlying block-diagonal structures of prior covariance matrices. To address intercluster connectivity, two alternative data adaptations are proposed. Once the localization is completed, a covariance diagnosis and tuning are performed within each cluster, whose contribution is sequentially integrated into the entire covariance matrix. Numerical twin experiment tests show the reduced cost and added flexibility of this approach compared to global covariance tuning, and more accurate results yielded for both observation- and background-error parameter tuning.

[1]  Trevor Hastie,et al.  The Elements of Statistical Learning , 2001 .

[2]  G. Evensen Sequential data assimilation with a nonlinear quasi‐geostrophic model using Monte Carlo methods to forecast error statistics , 1994 .

[3]  Anthony M. Castronova,et al.  A hierarchical network-based algorithm for multi-scale watershed delineation , 2014, Comput. Geosci..

[4]  A. Lorenc,et al.  Operational implementation of a hybrid ensemble/4D‐Var global data assimilation system at the Met Office , 2013 .

[5]  M. Buehner,et al.  Scale-dependent background-error covariance localisation , 2015 .

[6]  E. Cuthill,et al.  Reducing the bandwidth of sparse symmetric matrices , 1969, ACM '69.

[7]  Louis J. Durlofsky,et al.  A New Differentiable Parameterization Based on Principal Component Analysis for the Low-Dimensional Representation of Complex Geological Models , 2014, Mathematical Geosciences.

[8]  A. Weaver,et al.  Representation of correlation functions in variational assimilation using an implicit diffusion operator , 2010 .

[9]  G. Evensen,et al.  Data assimilation in the geosciences: An overview of methods, issues, and perspectives , 2017, WIREs Climate Change.

[10]  Don Coppersmith,et al.  Matrix multiplication via arithmetic progressions , 1987, STOC.

[11]  B. Iooss,et al.  Error covariance tuning in variational data assimilation: application to an operating hydrological model , 2020, Stochastic Environmental Research and Risk Assessment.

[12]  E. Kalnay,et al.  Balance and Ensemble Kalman Filter Localization Techniques , 2011 .

[13]  S. Dance,et al.  On diagnosing observation‐error statistics with local ensemble data assimilation , 2017 .

[14]  P. Courtier,et al.  Correlation modelling on the sphere using a generalized diffusion equation , 2001 .

[15]  Christopher C. Pain,et al.  Optimal reduced space for Variational Data Assimilation , 2019, J. Comput. Phys..

[16]  Jean-Philippe Argaud,et al.  Variational assimilation for xenon dynamical forecasts in neutronic using advanced background error covariance matrix modelling , 2013, 1304.6836.

[17]  Qing Li,et al.  An inverse-distance-based fitting term for 3D-Var data assimilation in nuclear core simulation , 2020 .

[18]  S. Cohn,et al.  Ooce Note Series on Global Modeling and Data Assimilation Construction of Correlation Functions in Two and Three Dimensions and Convolution Covariance Functions , 2022 .

[19]  Philippe Moireau,et al.  Front shape similarity measure for shape-oriented sensitivity analysis and data assimilation for Eikonal equation , 2018 .

[20]  Didier Lucor,et al.  Cardiovascular Modeling With Adapted Parametric Inference , 2018 .

[21]  Mark Buehner,et al.  Interchannel error correlation associated with AIRS radiance observations : Inference and impact in data assimilation , 2007 .

[22]  Ulises Cortés,et al.  Fluid Communities: A Competitive, Scalable and Diverse Community Detection Algorithm , 2017, COMPLEX NETWORKS.

[23]  T. Heckmann,et al.  Graph theory in the geosciences , 2015 .

[24]  N. Papadakis,et al.  Accounting for observation errors in image data assimilation , 2015 .

[25]  Christopher C. Pain,et al.  Optimal vaccination strategies for COVID-19 based on dynamical social networks with real-time updating , 2021, medRxiv.

[26]  Marc Bocquet,et al.  On the Efficiency of Covariance Localisation of the Ensemble Kalman Filter Using Augmented Ensembles , 2019, Front. Appl. Math. Stat..

[27]  Marc Bocquet,et al.  Advanced Data Assimilation for Geosciences: Lecture Notes of the Les Houches School of Physics: Special Issue, June 2012 , 2014 .

[28]  S. Srinivasan,et al.  Indicator-based data assimilation with multiple-point statistics for updating an ensemble of models with non-Gaussian parameter distributions , 2020 .

[29]  P. Leeuwen Non-local Observations and Information Transfer in Data Assimilation , 2019, Frontiers Appl. Math. Stat..

[30]  K. Haines,et al.  Using lagged covariances in data assimilation , 2017 .

[31]  M. Gerke Using horizontal and vertical building structure to constrain indirect sensor orientation , 2011 .

[32]  Arnaud Browet,et al.  Low-rank Similarity Measure for Role Model Extraction , 2013, ArXiv.

[33]  Nancy Nichols,et al.  Data assimilation with correlated observation errors: experiments with a 1-D shallow water model , 2013 .

[34]  Behnam Jafarpour,et al.  A Probability Conditioning Method (PCM) for Nonlinear Flow Data Integration into Multipoint Statistical Facies Simulation , 2011 .

[35]  Yingrui Yu,et al.  Reactor power distribution detection and estimation via a stabilized gappy proper orthogonal decomposition method , 2020 .

[36]  Don Coppersmith,et al.  Matrix multiplication via arithmetic progressions , 1987, STOC.

[37]  Florence Rabier,et al.  Properties and first application of an error‐statistics tuning method in variational assimilation , 2004 .

[38]  Eugenia Kalnay,et al.  Correlation-Cutoff Method for Covariance Localization in Strongly Coupled Data Assimilation , 2018, Monthly Weather Review.

[39]  Paul Poli,et al.  Diagnosis of observation, background and analysis‐error statistics in observation space , 2005 .

[40]  Istvan Szunyogh,et al.  Efficient data assimilation for spatiotemporal chaos: A local ensemble transform Kalman filter , 2005, physics/0511236.

[41]  Padhraic Smyth,et al.  Graphical models for statistical inference and data assimilation , 2007, Physica D: Nonlinear Phenomena.

[42]  H. Kim,et al.  Stream gauge network grouping analysis using community detection , 2020, Stochastic Environmental Research and Risk Assessment.

[43]  Jean-Loup Guillaume,et al.  Fast unfolding of communities in large networks , 2008, 0803.0476.

[44]  Adrian Sandu,et al.  AN ERROR SUBSPACE PERSPECTIVE ON DATA ASSIMILATION , 2015 .

[45]  Robert Atlas,et al.  Error Estimates for Ocean Surface Winds: Applying Desroziers Diagnostics to the Cross-Calibrated, Multiplatform Analysis of Wind Speed , 2013 .

[46]  Róbert Busa-Fekete,et al.  GraphClus, a MATLAB program for cluster analysis using graph theory , 2009, Comput. Geosci..

[47]  Yike Guo,et al.  A Reduced Order Deep Data Assimilation model , 2020 .

[48]  David J. Ketchen,et al.  THE APPLICATION OF CLUSTER ANALYSIS IN STRATEGIC MANAGEMENT RESEARCH: AN ANALYSIS AND CRITIQUE , 1996 .

[49]  John Derber,et al.  The National Meteorological Center's spectral-statistical interpolation analysis system , 1992 .

[50]  Thierry Chonavel,et al.  Estimating model‐error covariances in nonlinear state‐space models using Kalman smoothing and the expectation–maximization algorithm , 2017 .

[51]  Robert Pincus,et al.  Parameter estimation using data assimilation in an atmospheric general circulation model: From a perfect toward the real world , 2013 .

[52]  Jean-Philippe Argaud,et al.  Background error covariance iterative updating with invariant observation measures for data assimilation , 2019, Stochastic Environmental Research and Risk Assessment.

[53]  Santo Fortunato,et al.  Community detection in graphs , 2009, ArXiv.

[54]  Nancy Nichols,et al.  Diagnosing Observation Error Correlations for Doppler Radar Radial Winds in the Met Office UKV Model Using Observation-Minus-Background and Observation-Minus-Analysis Statistics , 2016 .

[55]  Gérald Desroziers,et al.  Diagnosis and adaptive tuning of observation‐error parameters in a variational assimilation , 2001 .

[56]  S. Srinivasan,et al.  Ensemble-Based Assimilation of Nonlinearly Related Dynamic Data in Reservoir Models Exhibiting Non-Gaussian Characteristics , 2018, Mathematical Geosciences.

[57]  Robert Tibshirani,et al.  Estimating the number of clusters in a data set via the gap statistic , 2000 .

[58]  Jean-Charles Delvenne,et al.  Rock-paper-scissors dynamics from random walks on temporal multiplex networks , 2020, J. Complex Networks.