Factor extraction using Kalman filter and smoothing: This is not just another survey

Abstract Dynamic factor models have been the main “big data” tool used by empirical macroeconomists during the last 30 years. In this context, Kalman filter and smoothing (KFS) procedures can cope with missing data, mixed frequency data, time-varying parameters, non-linearities, non-stationarity, and many other characteristics often observed in real systems of economic variables. The main contribution of this paper is to provide a comprehensive updated summary of the literature on latent common factors extracted using KFS procedures in the context of dynamic factor models, pointing out their potential limitations. Signal extraction and parameter estimation issues are separately analyzed. Identification issues are also tackled in both stationary and non-stationary models. Finally, empirical applications are surveyed in both cases. This survey is relevant to researchers and practitioners interested not only in the theory of KFS procedures for factor extraction in dynamic factor models but also in their empirical application in macroeconomics and finance.

[1]  Monetary Policy in Real Time , 2004, NBER Macroeconomics Annual.

[3]  Anindya Banerjee,et al.  Structural FECM: Cointegration in Large-Scale Structural FAVAR Models , 2014 .

[4]  R. Engle,et al.  A One-Factor Multivariate Time Series Model of Metropolitan Wage Rates , 1981 .

[5]  Jörg Breitung,et al.  A Canonical Correlation Approach for Selecting the Number of Dynamic Factors , 2013 .

[6]  G. Chow.  Analysis and control of dynamic economic systems , 1975 .

[7]  Seung C. Ahn,et al.  Eigenvalue Ratio Test for the Number of Factors , 2013 .

[8]  Byeongchan Seong,et al.  Estimation of vector error correction models with mixed‐frequency data , 2013 .

[9]  Yasutomo Murasawa,et al.  A Coincident Index, Common Factors, and Monthly Real GDP , 2010 .

[10]  Daniel Peña,et al.  Statistical Learning for Big Dependent Data , 2021 .

[11]  Daniel Peña,et al.  COINTEGRATION AND COMMON FACTORS , 1994 .

[12]  C. Whiteman,et al.  International Business Cycles: World, Region, and Country-Specific Factors , 2003 .

[13]  F. Diebold,et al.  Forecasting the Term Structure of Government Bond Yields , 2002 .

[14]  Matteo Barigozzi,et al.  Common Factors, Trends, and Cycles in Large Datasets , 2017, 1709.01445.

[15]  Gerhard Rünstler,et al.  On the Design of Data Sets for Forecasting with Dynamic Factor Models , 2010, SSRN Electronic Journal.

[16]  Peng Wang,et al.  Econometric Analysis of Large Factor Models , 2016 .

[17]  S. Koopman,et al.  Disturbance smoother for state space models , 1993 .

[18]  Brandyn Bok,et al.  Macroeconomic Nowcasting and Forecasting with Big Data , 2017 .

[19]  Daniel Peña,et al.  Forecasting Multiple Time Series With One-Sided Dynamic Principal Components , 2017, Journal of the American Statistical Association.

[20]  Enzo Weber,et al.  On the identification of multivariate correlated unobserved components models , 2016 .

[21]  Máximo Camacho,et al.  Can we use seasonally adjusted variables in dynamic factor models? , 2014 .

[22]  J. Bai,et al.  Large Dimensional Factor Analysis , 2008 .

[23]  Davide Delle Monache,et al.  Efficient matrix approach for classical inference in state space models , 2019 .

[24]  Siem Jan Koopman,et al.  Forecasting Macroeconomic Variables Using Collapsed Dynamic Factor Analysis , 2012 .

[25]  S. Koopman,et al.  Analyzing the Term Structure of Interest Rates Using the Dynamic Nelson–Siegel Model With Time-Varying Parameters , 2010 .

[26]  New York Dover,et al.  ON THE CONVERGENCE PROPERTIES OF THE EM ALGORITHM , 1983 .

[27]  David H. Small,et al.  Nowcasting: the real time informational content of macroeconomic data releases , 2008 .

[28]  P. D. Jong.  The Diffuse Kalman Filter , 1991 .

[29]  Glenn D. Rudebusch,et al.  The Macroeconomy and the Yield Curve: A Nonstructural Analysis , 2003 .

[30]  M. Hallin,et al.  Determining the Number of Factors in the General Dynamic Factor Model , 2007 .

[31]  M. Barigozzi,et al.  Quasi Maximum Likelihood Estimation of Non-Stationary Large Approximate Dynamic Factor Models , 2019, 1910.09841.

[32]  J. Bai,et al.  Principal components estimation and identification of static factors , 2013 .

[33]  F. Diebold.  Real-Time Real Economic Activity: Exiting the Great Recession and Entering the Pandemic Recession , 2020, SSRN Electronic Journal.

[34]  Xiaowen Jin,et al.  Forecasting and Nowcasting Real GDP: Comparing Statistical Models and Subjective Forecasts , 2012 .

[35]  N. Shephard,et al.  Exact score for time series models in state space form , 1992 .

[36]  D. Peña,et al.  Forecasting with nonstationary dynamic factor models , 2004 .

[37]  Enrique Sentana,et al.  Normality tests for latent variables , 2019, Quantitative Economics.

[38]  R. Shumway,et al.  AN APPROACH TO TIME SERIES SMOOTHING AND FORECASTING USING THE EM ALGORITHM , 1982 .

[39]  Xiao-Li Meng,et al.  Using EM to Obtain Asymptotic Variance-Covariance Matrices: The SEM Algorithm , 1991 .

[40]  J. Bai,et al.  A Panic Attack on Unit Roots and Cointegration , 2001 .

[41]  Forecas,et al.  Forecasting with Approximate Dynamic Factor Models : the role of non-pervasive shocks , 2011 .

[42]  Massimiliano Marcellino,et al.  EUROMIND: a monthly indicator of the euro area economic conditions , 2011 .

[43]  Pilar Poncela,et al.  Estimating Non-stationary Common Factors: Implications for Risk Sharing , 2018, Computational Economics.

[44]  C. Schumacher,et al.  Bayesian estimation of sparse dynamic factor models with order-independent and ex-post mode identification , 2019, Journal of Econometrics.

[45]  Jaime Martinez-Martin,et al.  Keeping track of global trade in real time , 2021 .

[46]  G. Mesters,et al.  On the Demographic Adjustment of Unemployment , 2017, Review of Economics and Statistics.

[47]  Riccardo Cristadoro,et al.  Short-Term Forecasting of GDP Using Large Monthly Datasets – A Pseudo Real-Time Forecast Evaluation Exercise , 2008 .

[48]  Jushan Bai,et al.  Estimating cross-section common stochastic trends in nonstationary panel data , 2004 .

[49]  Massimiliano Marcellino,et al.  A comparison of mixed frequency approaches for nowcasting Euro area macroeconomic aggregates , 2014 .

[50]  A spectral EM algorithm for dynamic factor models , 2018, Journal of Econometrics.

[51]  Vasja Sivec,et al.  Monetary, Fiscal and Oil Shocks: Evidence Based on Mixed Frequency Structural FAVARs , 2015 .

[52]  Nowcasting Irish GDP , 2013 .

[53]  R. Jennrich,et al.  Standard errors for EM estimation , 2000 .

[54]  E. Hannan,et al.  The statistical theory of linear systems , 1989 .

[55]  Pilar Poncela,et al.  A comment on the dynamic factor model with dynamic factors , 2020 .

[56]  José Antonio Ortega,et al.  Joint forecasts of Southern European fertility rates with non-stationary dynamic factor models , 2005 .

[57]  Marcelle Chauvet,et al.  A dynamic factor model of the yield curve components as a predictor of the economy , 2016 .

[58]  Maximo Camacho,et al.  Markov-Switching Dynamic Factor Models in Real Time , 2012, International Journal of Forecasting.

[59]  Siem Jan Koopman,et al.  Likelihood�?Based Dynamic Factor Analysis for Measurement and Forecasting , 2015 .

[60]  Troy D. Matheson.  Financial Conditions Indexes for the United States and Euro Area , 2011, SSRN Electronic Journal.

[61]  Pilar Poncela,et al.  The Effects of Disaggregation on Forecasting Nonstationary Time Series , 2014 .

[62]  R. Cattell.  The Scree Test For The Number Of Factors. , 1966, Multivariate behavioral research.

[63]  Estimating latent asset-pricing factors , 2020 .

[64]  S. Koopman,et al.  Forecasting and nowcasting economic growth in the euro area using factor models , 2016 .

[65]  S. Koopman,et al.  Forecasting the U.S. Term Structure of Interest Rates Using a Macroeconomic Smooth Dynamic Factor Model , 2012 .

[66]  S. Mittnik.  Macroeconomic Forecasting Using Pooled International Data , 1990 .

[67]  Danny Quah,et al.  A Dynamic Index Model for Large Cross Sections , 1993 .

[68]  Domenico Giannone,et al.  Unspanned Macroeconomic Factors in the Yield Curve , 2014 .

[69]  Robert H. Shumway,et al.  On computing the expected Fisher information matrix for state-space model parameters , 1996 .

[70]  Pilar Poncela,et al.  Determining the number of factors after stationary univariate transformations , 2016, Empirical Economics.

[71]  Real-Time Measurement of Business Conditions , 2009 .

[72]  Catherine Doz,et al.  A Quasi–Maximum Likelihood Approach for Large, Approximate Dynamic Factor Models , 2006, Review of Economics and Statistics.

[73]  Michael A. West,et al.  BAYESIAN MODEL ASSESSMENT IN FACTOR ANALYSIS , 2004 .

[74]  Ian T. Jolliffe,et al.  Estimating common trends in multivariate time series using dynamic factor analysis , 2003 .

[75]  Michael P. Clements,et al.  Dynamic Factor Models , 2011, Financial Econometrics.

[76]  Kunpeng Li,et al.  Maximum Likelihood Estimation and Inference for Approximate Factor Models of High Dimension , 2016, Review of Economics and Statistics.

[77]  Rodney W. Strachan,et al.  Invariant Inference and Efficient Computation in the Static Factor Model , 2013 .

[78]  Andrew Harvey,et al.  TESTS OF COMMON STOCHASTIC TRENDS , 2000, Econometric Theory.

[79]  Glenn D. Rudebusch,et al.  The Macroeconomy and the Yield Curve: A Dynamic Latent Factor Approach , 2004 .

[80]  Marta Bańbura,et al.  A Look into the Factor Model Black Box: Publication Lags and the Role of Hard and Soft Data in Forecasting GDP , 2007, SSRN Electronic Journal.

[81]  G. Koop,et al.  A New Index of Financial Conditions , 2013 .

[82]  George Kapetanios,et al.  Forecasting UK inflation bottom up , 2021, SSRN Electronic Journal.

[83]  Clifford Lam,et al.  Factor modeling for high-dimensional time series: inference for the number of factors , 2012, 1206.0613.

[84]  Mark W. Watson,et al.  Dynamic Factor Models, Factor-Augmented Vector Autoregressions, and Structural Vector Autoregressions in Macroeconomics , 2016 .

[85]  Herman Rubin,et al.  Statistical Inference in Factor Analysis , 1956 .

[86]  D. Giannone,et al.  Now-Casting and the Real-time Data Flow , 2012, SSRN Electronic Journal.

[87]  S. Johansen,et al.  Cointegration between Trends and Their Estimators in State Space Models and Cointegrated Vector Autoregressive Models , 2017 .

[88]  Marc K. Francke,et al.  Likelihood functions for state space models with diffuse initial conditions , 2006 .

[89]  Fabio H. Nieto,et al.  Choosing a dynamic common factor as a coincident index , 2016 .

[90]  Máximo Camacho,et al.  Toward a more reliable picture of the economic activity: An application to Argentina , 2015 .

[91]  Helmut Ltkepohl,et al.  New Introduction to Multiple Time Series Analysis , 2007 .

[92]  Catherine Doz,et al.  A Two-Step Estimator for Large Approximate Dynamic Factor Models Based on Kalman Filtering , 2007 .

[93]  N. Shephard,et al.  Multivariate stochastic variance models , 1994 .

[94]  Borus Jungbacker,et al.  Maximum Likelihood Estimation for Dynamic Factor Models with Missing Data , 2011 .

[95]  J. Bai,et al.  Inferential Theory for Factor Models of Large Dimensions , 2003 .

[96]  Pilar Poncela.  Comments on “Forecasting the US term structure of interest rates using a macroeconomic smooth dynamic factor model” by Koopman and van der Wel , 2013 .

[97]  Leif Anders Thorsrud.  Words are the New Numbers: A Newsy Coincident Index of the Business Cycle , 2018, Journal of Business & Economic Statistics.

[98]  Mark W. Watson,et al.  Dynamic Factor Models: A Brief Retrospective , 2016 .

[99]  Gabriele Fiorentini,et al.  Likelihood-based estimation of latent generalised ARCH structures , 2004 .

[100]  Olivier Darné,et al.  Dynamic Factor Models: A Review of the Literature , 2013 .

[101]  R. Engle,et al.  Alternative Algorithms for the Estimation of Dynamic Factor , 1983 .

[102]  R. Kohn,et al.  Estimation, Filtering, and Smoothing in State Space Models with Incompletely Specified Initial Conditions , 1985 .

[103]  Domenico Giannone,et al.  Comparing Alternative Predictors Based on Large‐Panel Factor Models , 2006 .

[104]  Massimiliano Marcellino,et al.  Factor Midas for Nowcasting and Forecasting with Ragged-Edge Data: A Model Comparison for German GDP , 2008 .

[105]  Marco Lippi,et al.  Cointegration and Error Correction Mechanisms for Singular Stochastic Vectors , 2020, Econometrics.

[106]  Fabio H. Nieto,et al.  Common seasonality in multivariate time series , 2016 .

[107]  Michele Modugno,et al.  Maximum Likelihood Estimation of Factor Models on Data Sets with Arbitrary Pattern of Missing Data , 2010, SSRN Electronic Journal.

[108]  A. Onatski.  TESTING HYPOTHESES ABOUT THE NUMBER OF FACTORS IN LARGE FACTOR MODELS , 2009 .

[109]  Martin Solberger,et al.  Estimating a Dynamic Factor Model in EViews Using the Kalman Filter and Smoother , 2019, Computational Economics.

[110]  Tommaso Proietti,et al.  Estimation of Common Factors under Cross‐Sectional and Temporal Aggregation Constraints , 2011 .

[111]  Michael T. Owyang,et al.  Specification and Estimation of Bayesian Dynamic Factor Models: A Monte Carlo Analysis with an Application to Global House Price Comovement , 2015 .

[112]  Andrew Harvey,et al.  Forecasting, structural time series models and the Kalman filter: Selected answers to exercises , 1990 .

[113]  Carolina García-Martos,et al.  Seasonal Dynamic Factor Analysis and Bootstrap Inference: Application to Electricity Market Forecasting , 2011, Technometrics.

[114]  Short-term forecasting for empirical economists. A survey of the recently proposed algorithms , 2013 .

[115]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[116]  Knut Are Aastveit,et al.  Nowcasting norwegian GDP: the role of asset prices in a small open economy , 2012 .

[117]  Marco Lippi,et al.  The generalized dynamic factor model: consistency and rates , 2004 .

[118]  Gabriel Pérez-Quirós,et al.  Country Shocks, Monetary Policy Expectations and ECB Decisions. A Dynamic Non-Linear Approach , 2015 .

[119]  M. Hallin,et al.  Dynamic factor models with infinite-dimensional factor spaces: One-sided representations , 2013 .

[120]  M. Haugh,et al.  Scenario analysis for derivative portfolios via dynamic factor models , 2020 .

[121]  Maximo Camacho,et al.  Real-time forecasting US GDP from small-scale factor models , 2014 .

[122]  A. García-Ferrer,et al.  Evaluating early warning and coincident indicators of business cycles using smooth trends , 2019, Journal of Forecasting.

[123]  Scott A. Brave,et al.  Diagnosing the Financial System: Financial Conditions and Financial Stress , 2012 .

[124]  M. Hallin,et al.  The Generalized Dynamic-Factor Model: Identification and Estimation , 2000, Review of Economics and Statistics.

[125]  A. García-Ferrer,et al.  Predicting Recessions with Factor Linear Dynamic Harmonic Regressions , 2013 .

[126]  Christian R. Proaño,et al.  Dissecting the financial cycle with dynamic factor models , 2017 .

[127]  A. Harvey,et al.  Diagnostic Checking of Unobserved-Components Time Series Models , 1992 .

[128]  Thomas J. Sargent,et al.  Business cycle modeling without pretending to have too much a priori economic theory , 1976 .

[129]  Gabriel Pérez-Quirós,et al.  Extracting Nonlinear Signals from Several Economic Indicators , 2012 .

[130]  J. Bai,et al.  Determining the Number of Factors in Approximate Factor Models , 2000 .

[131]  In Choi,et al.  Model selection for factor analysis: Some new criteria and performance comparisons , 2019 .

[132]  W. Jos Jansen,et al.  Combining Model‐Based Near‐Term GDP Forecasts and Judgmental Forecasts: A Real‐Time Exercise for the G7 Countries , 2018, Oxford Bulletin of Economics and Statistics.

[133]  F. Diebold,et al.  Real-Time Macroeconomic Monitoring: Real Activity, Inflation, and Interactions , 2010 .

[134]  Matteo Barigozzi,et al.  Improved penalization for determining the number of factors in approximate factor models , 2010 .

[135]  Matteo Luciani,et al.  Large-Dimensional Dynamic Factor Models in Real-Time: A Survey , 2014 .

[136]  George Kapetanios,et al.  A parametric estimation method for dynamic factor models of large dimensions , 2006 .

[137]  M. Hallin,et al.  Dynamic Factor Models with Infinite-Dimensional Factor Space: Asymptotic Analysis , 2015 .

[138]  J. Stock,et al.  Forecasting Using Principal Components From a Large Number of Predictors , 2002 .

[139]  Matteo Luciani.  Monetary Policy and the Housing Market: A Structural Factor Analysis , 2015 .

[140]  D. Giannone,et al.  Common Factors of Commodity Prices , 2017, Journal of Applied Econometrics.

[141]  Troy D. Matheson.  An analysis of the informational content of New Zealand data releases: The importance of business opinion surveys , 2007 .

[142]  A Monte Carlo comparison of estimating the number of dynamic factors , 2017 .

[143]  J. Stock,et al.  Testing for Common Trends , 1988 .

[144]  Matteo Barigozzi,et al.  Quasi Maximum Likelihood Estimation and Inference of Large Approximate Dynamic Factor Models via the EM algorithm , 2019, 1910.03821.

[145]  Massimiliano Marcellino,et al.  A comparison of methods for the construction of composite coincident and leading indexes for the UK , 2007 .

[146]  Piet de Jong,et al.  The likelihood for a state space model , 1988 .

[147]  Carl S. Bonham,et al.  FORECASTING WITH MIXED-FREQUENCY FACTOR MODELS IN THE PRESENCE OF COMMON TRENDS , 2013, Macroeconomic Dynamics.

[148]  Mark W. Watson,et al.  Consistent Estimation of the Number of Dynamic Factors in a Large N and T Panel , 2007 .

[149]  M. Marcellino,et al.  EuroMInd-C: A Disaggregate Monthly Indicator of Economic Activity for the Euro Area and Member Countries , 2013 .

[150]  A. D. Reijer,et al.  Nowcasting Swedish GDP with a large and unbalanced data set , 2018, Empirical Economics.

[151]  Kunpeng Li,et al.  STATISTICAL ANALYSIS OF FACTOR MODELS OF HIGH DIMENSION , 2012, 1205.6617.

[152]  Mark W. Watson,et al.  Twenty Years of Time Series Econometrics in Ten Pictures , 2017 .

[153]  Pilar Poncela,et al.  Global vs Sectoral Factors and the Impact of the Financialization in Commodity Price Changes , 2020, Open Economies Review.

[154]  Damián López,et al.  Multivariate Exponential Smoothing and Dynamic Factor Model Applied to Hourly Electricity Price Analysis , 2014, Technometrics.

[155]  R. Mariano,et al.  A New Coincident Index of Business Cycles Based on Monthly and Quarterly Series , 2002 .

[156]  A. Harvey,et al.  Multivariate structural time series models , 1997 .

[157]  Chiara Scotti.  Surprise and Uncertainty Indexes: Real-Time Aggregation of Real-Activity Macro Surprises , 2016 .

[158]  John Geweke,et al.  Maximum Likelihood "Confirmatory" Factor Analysis of Economic Time Series , 1981 .

[159]  A. Onatski.  Determining the Number of Factors from Empirical Distribution of Eigenvalues , 2010, The Review of Economics and Statistics.

[160]  Melvin J. Hinich,et al.  Time Series Analysis by State Space Methods , 2001 .

[161]  Pilar Poncela,et al.  Forecasting European GNP data through common factor models and other procedures , 2002 .

[162]  Gonzalo Camba-Mendez,et al.  Short-Term Forecasts of Euro Area GDP Growth , 2008, SSRN Electronic Journal.

[163]  D. Oakes.  Direct calculation of the information matrix via the EM , 1999 .

[164]  S. Koopman,et al.  SMOOTH DYNAMIC FACTOR ANALYSIS WITH APPLICATION TO THE US TERM STRUCTURE OF INTEREST RATES: SMOOTH DYNAMIC FACTOR ANALYSIS , 2014 .

[165]  Máximo Camacho,et al.  Short-Run Forecasting of Argentine Gross Domestic Product Growth , 2015 .

[166]  Charles H. Whiteman,et al.  Bayesian Leading Indicators: Measuring and Predicting Economic Conditions in Iowa , 1998 .

[167]  S. Koopman,et al.  Exact Initial Kalman Filtering and Smoothing for Nonstationary Time Series Models , 1997 .

[168]  Catherine Doz,et al.  Forecasting French GDP with Dynamic Factor Models : a pseudo-real time experiment using Factor-augmented Error Correction Models , 2018 .

[169]  Daniel Peña,et al.  Nonstationary dynamic factor analysis , 2006 .

[170]  Simon M. Potter,et al.  Dynamic Hierarchical Factor Models , 2011, Review of Economics and Statistics.

[171]  Christian Schumacher,et al.  POOLING VERSUS MODEL SELECTION FOR NOWCASTING GDP WITH MANY PREDICTORS: EMPIRICAL EVIDENCE FOR SIX INDUSTRIALIZED COUNTRIES , 2013 .

[172]  Massimiliano Marcellino,et al.  Classical time varying factor‐augmented vector auto‐regressive models—estimation, forecasting and structural analysis , 2015 .

[173]  Siem Jan Koopman,et al.  Computing Observation Weights for Signal Extraction and Filtering , 2003 .

[174]  Pilar Poncela,et al.  More is not always better : back to the Kalman lter in Dynamic Factor Models , 2012 .

[175]  Q. Yao,et al.  Modelling multiple time series via common factors , 2008 .

[176]  J. Stock,et al.  A Probability Model of the Coincident Economic Indicators , 1988 .

[177]  Stephan Smeekes,et al.  A dynamic factor model approach to incorporate Big Data in state space models for official statistics , 2019, Journal of the Royal Statistical Society: Series A (Statistics in Society).

[178]  Gabriel Pérez-Quirós,et al.  Disentangling Contagion Among Sovereign CDS Spreads During the European Debt Crisis , 2013 .

[179]  Maximo Camacho,et al.  Introducing the Euro-Sting: Short-Term Indicator of Euro Area Growth , 2009 .

[180]  D. Stoffer,et al.  Bootstrapping State-Space Models: Gaussian Maximum Likelihood Estimation and the Kalman Filter , 1991 .

[181]  Gabriele Fiorentini,et al.  Dynamic specification tests for dynamic factor models , 2019, Journal of Applied Econometrics.

[182]  Serena Ng,et al.  Determining the Number of Primitive Shocks in Factor Models , 2007 .

[183]  Marco Lippi,et al.  The Generalized Dynamic Factor Model , 2002 .

[184]  António Rua,et al.  Dynamic Factor Models with Jagged Edge Panel Data: Taking on Board the Dynamics of the Idiosyncratic Components , 2013 .

[185]  Sandra Eickmeier,et al.  How Successful are Dynamic Factor Models at Forecasting Output and Inflation? A Meta-Analytic Approach , 2008 .