A Hybrid MPI–OpenMP Parallel Algorithm and Performance Analysis for an Ensemble Square Root Filter Designed for Multiscale Observations

AbstractA hybrid parallel scheme for the ensemble square root filter (EnSRF) suitable for parallel assimilation of multiscale observations, including those from dense observational networks such as those of radar, is developed based on the domain decomposition strategy. The scheme handles internode communication through a message passing interface (MPI) and the communication within shared-memory nodes via Open Multiprocessing (OpenMP) threads. It also supports pure MPI and pure OpenMP modes. The parallel framework can accommodate high-volume remote-sensed radar (or satellite) observations as well as conventional observations that usually have larger covariance localization radii.The performance of the parallel algorithm has been tested with simulated and real radar data. The parallel program shows good scalability in pure MPI and hybrid MPI–OpenMP modes, while pure OpenMP runs exhibit limited scalability on a symmetric shared-memory system. It is found that in MPI mode, better parallel performance is achi...

[1]  Guifu Zhang,et al.  State estimation of convective storms with a two-moment microphysics scheme and an ensemble Kalman filter: Experiments with simulated radar data , 2010 .

[2]  Jidong Gao,et al.  The Advanced Regional Prediction System (ARPS), storm-scale numerical weather prediction and data assimilation , 2003 .

[3]  Istvan Szunyogh,et al.  A local ensemble transform Kalman filter data assimilation system for the NCEP global model , 2008 .

[4]  M. Xue,et al.  Analysis of a Tornadic Mesoscale Convective Vortex Based on Ensemble Kalman Filter Assimilation of CASA X-Band and WSR-88D Radar Data , 2011 .

[5]  Ming Xue,et al.  The analysis and impact of simulated high-resolution surface observations in addition to radar data for convective storms with an ensemble Kalman filter , 2011 .

[6]  M. Xue,et al.  The Advanced Regional Prediction System (ARPS) – A multi-scale nonhydrostatic atmospheric simulation and prediction tool. Part II: Model physics and applications , 2001 .

[7]  Ming Xue,et al.  A four‐dimensional asynchronous ensemble square‐root filter (4DEnSRF) algorithm and tests with simulated radar data , 2013 .

[8]  Ming Xue,et al.  Numerical Prediction of High-Impact Local Weather: A Driver for Petascale Computing , 2007 .

[9]  Chris Snyder,et al.  A Comparison between the 4DVAR and the Ensemble Kalman Filter Techniques for Radar Data Assimilation , 2005 .

[10]  M. Rienecker,et al.  Initial testing of a massively parallel ensemble Kalman filter with the Poseidon isopycnal ocean general circulation model , 2002 .

[11]  Xiaolin Li,et al.  Hybrid Runtime Management of Space-Time Heterogeneity for Parallel Structured Adaptive Applications , 2007, IEEE Transactions on Parallel and Distributed Systems.

[12]  Mingjing Tong,et al.  An OSSE Framework Based on the Ensemble Square Root Kalman Filter for Evaluating the Impact of Data from Radar Networks on Thunderstorm Analysis and Forecasting , 2006 .

[13]  Mingjing Tong,et al.  Simultaneous state estimation and attenuation correction for thunderstorms with radar data using an ensemble Kalman filter: tests with simulated data , 2009 .

[14]  Manish Parashar,et al.  Advanced Computational Infrastructures for Parallel and Distributed Adaptive Applications , 2009 .

[15]  G. Evensen,et al.  Asynchronous data assimilation with the EnKF , 2010 .

[16]  Craig H. Bishop,et al.  Adaptive sampling with the ensemble transform Kalman filter , 2001 .

[17]  Juanzhen Sun,et al.  Dynamical and Microphysical Retrieval from Doppler Radar Observations Using a Cloud Model and Its Adjoint. Part I: Model Development and Simulated Data Experiments. , 1997 .

[18]  C. Snyder,et al.  Assimilation of Simulated Doppler Radar Observations with an Ensemble Kalman Filter , 2003 .

[19]  Bing Wu,et al.  Dynamical and Microphysical Retrievals from Doppler Radar Observations of a Deep Convective Cloud , 2000 .

[20]  Juanzhen Sun,et al.  Dynamical and Microphysical Retrieval from Doppler Radar Observations Using a Cloud Model and Its Adjoint. Part II: Retrieval Experiments of an Observed Florida Convective Storm , 1998 .

[21]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[22]  Mingjing Tong,et al.  Ensemble Kalman Filter Analyses of the 29–30 May 2004 Oklahoma Tornadic Thunderstorm Using One- and Two-Moment Bulk Microphysics Schemes, with Verification against Polarimetric Radar Data , 2012 .

[23]  P. Houtekamer,et al.  Data Assimilation Using an Ensemble Kalman Filter Technique , 1998 .

[24]  Mingjing Tong,et al.  Simultaneous Estimation of Microphysical Parameters and Atmospheric State with Simulated Radar Data and Ensemble Square Root Kalman Filter. Part II: Parameter Estimation Experiments , 2008 .

[25]  Louis J. Wicker,et al.  Wind and Temperature Retrievals in the 17 May 1981 Arcadia, Oklahoma, Supercell: Ensemble Kalman Filter Experiments , 2004 .

[26]  Jerry M. Straka,et al.  Assimilation of Simulated Polarimetric Radar Data for a Convective Storm Using the Ensemble Kalman Filter. Part II: Impact of Polarimetric Data on Storm Analysis , 2008 .

[27]  M. Yau,et al.  A Multimoment Bulk Microphysics Parameterization. Part I: Analysis of the Role of the Spectral Shape Parameter , 2005 .

[28]  Ming Xue,et al.  Retrieval of Moisture from Slant-Path Water Vapor Observations of a Hypothetical GPS Network Using a Three-Dimensional Variational Scheme with Anisotropic Background Error , 2006 .

[29]  Jeffrey L. Anderson.  An Ensemble Adjustment Kalman Filter for Data Assimilation , 2001 .

[30]  G. Evensen,et al.  Analysis Scheme in the Ensemble Kalman Filter , 1998 .

[31]  M. Buehner,et al.  Intercomparison of Variational Data Assimilation and the Ensemble Kalman Filter for Global Deterministic NWP. Part II: One-Month Experiments with Real Observations , 2010 .

[32]  Ming Xue,et al.  Parallel Weather Modeling with the Advanced Regional Prediction System , 1997, Parallel Comput..

[33]  M. Xue,et al.  Analysis of a Tornadic Mesoscale Convective Vortex Based on Ensemble Kalman Filter Assimilation of CASA X-Band and WSR-88D Radar Data , 2011 .

[34]  F. L. Dimet,et al.  Variational algorithms for analysis and assimilation of meteorological observations: theoretical aspects , 1986 .

[35]  Jeffrey L. Anderson,et al.  Scalable Implementations of Ensemble Filter Algorithms for Data Assimilation , 2007 .

[36]  Robert B. Wilhelmson,et al.  The Morphology of Several Tornadic Storms on 20 May 1977 , 1981 .

[37]  K. Droegemeier,et al.  The Advanced Regional Prediction System (ARPS) – A multi-scale nonhydrostatic atmospheric simulation and prediction model. Part I: Model dynamics and verification , 2000 .

[38]  G. Evensen.  Sequential data assimilation with a nonlinear quasi‐geostrophic model using Monte Carlo methods to forecast error statistics , 1994 .

[39]  J. Whitaker,et al.  Ensemble Square Root Filters , 2003, Statistical Methods for Climate Scientists.

[40]  G. Evensen,et al.  Assimilation of Geosat altimeter data for the Agulhas current using the ensemble Kalman filter with , 1996 .

[41]  Stanley G. Benjamin,et al.  Global Ensemble Predictions of 2009’s Tropical Cyclones Initialized with an Ensemble Kalman Filter , 2011 .

[42]  Jeffrey L. Anderson.  A Local Least Squares Framework for Ensemble Filtering , 2003 .

[43]  Craig H. Bishop,et al.  Vertical Covariance Localization for Satellite Radiances in Ensemble Kalman Filters , 2010 .

[44]  Jimy Dudhia,et al.  The Weather Research and Forecast Model: software architecture and performance [presentation] , 2005 .

[45]  Mingjing Tong,et al.  Ensemble kalman filter assimilation of doppler radar data with a compressible nonhydrostatic model : OSS experiments , 2005 .

[46]  P. Courtier,et al.  Variational Assimilation of Meteorological Observations With the Adjoint Vorticity Equation. Ii: Numerical Results , 2007 .

[47]  Venkatramani Balaji,et al.  Initialization of an ENSO Forecast System Using a Parallelized Ensemble Filter , 2005 .

[48]  Geir Evensen,et al.  The Ensemble Kalman Filter: theoretical formulation and practical implementation , 2003 .

[49]  Louis J. Wicker,et al.  Ensemble Kalman Filter Assimilation of Radar Observations of the 8 May 2003 Oklahoma City Supercell: Influences of Reflectivity Observations on Storm-Scale Analyses , 2011 .

[50]  J. Whitaker,et al.  Ensemble Data Assimilation without Perturbed Observations , 2002 .