NBER WORKING PAPER SERIES EVALUATING CONTRADICTORY EXPERIMENTAL AND NON-EXPERIMENTAL ESTIMATES OF NEIGHBORHOOD EFFECTS ON ECONOMIC OUTCOMES FOR ADULTS

Although non-experimental studies find robust neighborhood effects on adults, such findings have been challenged by results from the Moving to Opportunity (MTO) residential mobility experiment. Using a within-study comparison design, this paper compares experimental and nonexperimental estimates from MTO and a parallel analysis of the Panel Study of Income Dynamics (PSID). Striking similarities were found between non-experimental estimates based on MTO and PSID. No clear evidence was found that different estimates are related to duration of adult exposure to disadvantaged neighborhoods, non-linear effects of neighborhood conditions, magnitude of the change in neighborhood context, frequency of moves, treatment effect heterogeneity, or measurement, although uncertainty bands around our estimates were sometimes large. One other possibility is that MTO-induced moves might have been unusually disruptive, but results are inconsistent for that hypothesis. Taken together, the findings suggest that selection bias might account for evidence of neighborhood effects on adult economic outcomes in nonexperimental studies. David J. Harding Department of Sociology 462 Social Science Building University of California, Berkeley Berkeley, CA 94720 dharding@berkeley.edu Lisa Sanbonmatsu Center for Education Policy Research at Harvard University 50 Church Street 4th Floor Cambridge, MA 02138 lsanbonm@nber.org Greg J. Duncan University of California, Irvine School of Education ́ 2056 Education Building, Mail Code 5500 ́Irvine, CA 92697 gduncan@uci.edu Lisa A. Gennetian Sanford School of Public Policy Duke University 212 Rubenstein Hall 302 Towerview Road Durham, NC 27708 United States lisa.gennetian@duke.edu Lawrence F. Katz Department of Economics Harvard University Cambridge, MA 02138 and NBER lkatz@harvard.edu Ronald C. Kessler Harvard Medical School Department of Health Care Policy 180 Longwood Avenue Boston, MA 02115 kessler@hcp.med.harvard.edu

[1]  Dionissi Aliprantis,et al.  Evidence of Neighborhood Effects from Moving to Opportunity: LATEs of Neighborhood Quality , 2019, Review of Economics and Statistics.

[2]  Sean F. Reardon,et al.  Has Income Segregation Really Increased? Bias and Bias Correction in Sample-Based Segregation Estimates , 2018, Demography.

[3]  Vivian C. Wong,et al.  What Can Be Learned From Empirical Evaluations of Nonexperimental Methods? , 2018, Evaluation review.

[4]  M. Lipsey,et al.  Can Propensity Score Analysis Approximate Randomized Experiments Using Pretest and Demographic Information in Pre-K Intervention Research? , 2018, Evaluation review.

[5]  D. Harding.  Evicted: Poverty and Profit in the American City , 2017 .

[6]  C. J. Gabbe,et al.  Employment proximity and outcomes for Moving to Opportunity families , 2017, Journal of urban affairs.

[7]  G. Galster.  People Versus Place, People and Place, or More? New Directions for Housing Policy , 2017 .

[8]  G. Galster,et al.  Do Neighborhood Effects on Low-Income Minority Children Depend on Their Age? Evidence From a Public Housing Natural Experiment , 2017, Housing policy debate.

[9]  Raj Chetty,et al.  The Impacts of Neighborhoods on Intergenerational Mobility Ii: County-Level Estimates , 2016 .

[10]  Raj Chetty,et al.  The Impacts of Neighborhoods on Intergenerational Mobility I: Childhood Exposure Effects , 2016 .

[11]  Eric T. Chyn.  Moved to Opportunity: The Long-Run Effect of Public Housing Demolition on Labor Market Outcomes of Children , 2016 .

[12]  Marie-Andree Somers,et al.  The Validity of the Comparative Interrupted Time Series Design for Evaluating the Effect of School-Level Interventions , 2016, Evaluation review.

[13]  Raj Chetty,et al.  The Effects of Exposure to Better Neighborhoods on Children: New Evidence from the Moving to Opportunity Experiment , 2015, The American economic review.

[14]  J. Angrist,et al.  Evaluating Econometric Evaluations of Post-Secondary Aid , 2015 .

[15]  J. Ludwig,et al.  The Impact of Housing Assistance on Child Outcomes: Evidence from a Randomized Housing Lottery , 2015 .

[16]  Philip M. Gleason,et al.  Horseshoes, Hand Grenades, and Treatment Effects? Reassessing Whether Nonexperimental Estimators are Biased , 2015 .

[17]  Qingyan Shang.  Endogenous neighborhood effects on welfare participation , 2014 .

[18]  P. Sharkey,et al.  Where, When, Why, and For Whom Do Residential Contexts Matter? Moving Away from the Dichotomous Understanding of Neighborhood Effects , 2014 .

[19]  Lawrence F. Katz,et al.  Associations of housing mobility interventions for children in high-poverty neighborhoods with subsequent mental disorders during adolescence. , 2014, JAMA.

[20]  E. Rosenbaum,et al.  Stuck in place: urban neighborhoods and the end of progress toward racial equality , 2014 .

[21]  P. Sharkey.  Spatial Segmentation and the Black Middle Class1 , 2014, American Journal of Sociology.

[22]  T. Pratt.  Great American city: Chicago and the enduring neighborhood effect , 2013 .

[23]  George Galster,et al.  Measuring Neighbourhood Effects Non-experimentally: How Much Do Alternative Methods Matter? , 2013 .

[24]  Lawrence F. Katz,et al.  Long-Term Neighborhood Effects on Low-Income Families: Evidence from Moving to Opportunity , 2013 .

[25]  Lawrence F. Katz,et al.  Neighborhood Effects on the Long-Term Well-Being of Low-Income Adults , 2012, Science.

[26]  R. Bifulco,et al.  CAN NONEXPERIMENTAL ESTIMATES REPLICATE ESTIMATES BASED ON RANDOM ASSIGNMENT IN EVALUATIONS OF SCHOOL CHOICE? A WITHIN-STUDY COMPARISON , 2012 .

[27]  K. Edin,et al.  Constrained Compliance: Solving the Puzzle of MTO's Lease-Up Rates and Why Mobility Matters , 2012 .

[28]  Greg J Duncan,et al.  Neighborhoods, obesity, and diabetes--a randomized social experiment. , 2011, The New England journal of medicine.

[29]  Scott J. South,et al.  SPATIAL AND TEMPORAL DIMENSIONS OF NEIGHBORHOOD EFFECTS ON HIGH SCHOOL GRADUATION. , 2011, Social science research.

[30]  Robert J. Sampson,et al.  Converging Evidence for Neighborhood Effects on Children's Test Scores: An Experimental, Quasi-experimental, and Observational Comparison* , 2010 .

[31]  Kevin Arceneaux,et al.  A Cautionary Note on the Use of Matching to Estimate Causal Effects: An Empirical Example Comparing Matching Estimates to an Experimental Benchmark , 2010 .

[32]  D. Harding.  Living the Drama: Community, Conflict, and Culture among Inner-City Boys , 2010 .

[33]  G. Duncan,et al.  Gautreaux mothers and their children: an update , 2010 .

[34]  Thomas D. Cook,et al.  Unbiased Causal Inference From an Observational Study: Results of a Within-Study Comparison , 2009 .

[35]  Thomas D. Cook,et al.  How Bias Reduction Is Affected by Covariate Choice, Unreliability, and Mode of Data Analysis: Results From Two Types of Within-Study Comparisons , 2009, Multivariate behavioral research.

[36]  D. Kirk.  A Natural Experiment on Residential Change and Recidivism: Lessons from Hurricane Katrina , 2009 .

[37]  Tara Watson,et al.  NBER WORKING PAPER SERIES INEQUALITY AND THE MEASUREMENT OF RESIDENTIAL SEGREGATION BY INCOME IN AMERICAN NEIGHBORHOODS , 2011 .

[38]  Jeffrey R. Kling,et al.  Effects of Neighborhood Characteristics on the Mortality of Black Male Youth: Evidence from Gautreaux Effects of Neighborhood Characteristics on the Mortality of Black Male Youth: Evidence from Gautreaux , 2004 .

[39]  Richard K. Crump,et al.  Dealing with limited overlap in estimation of average treatment effects , 2009 .

[40]  A. Kalleberg.  Precarious Work, Insecure Workers: Employment Relations in Transition , 2009 .

[41]  Bruce Western,et al.  Lone Pursuit: Distrust and Defensive Individualism Among the Black Poor , 2009 .

[42]  J. Ludwig,et al.  The Effects of Housing Assistance on Labor Supply: Evidence from a Voucher Lottery , 2012 .

[43]  Vivian C. Wong,et al.  Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within‐study comparisons , 2008 .

[44]  D. Massey,et al.  Neighborhood Effects on Economic Self‐Sufficiency: A Reconsideration of the Moving to Opportunity Experiment1 , 2008, American Journal of Sociology.

[45]  Robert J. Sampson,et al.  Moving to Inequality: Neighborhood Effects and Experiments Meet Social Structure1 , 2008, American Journal of Sociology.

[46]  D. Massey,et al.  Neighborhoods, employment, and welfare use: assessing the influence of neighborhood socioeconomic composition. , 2008, Social science research.

[47]  Robert J. Sampson,et al.  Durable effects of concentrated disadvantage on verbal ability among African-American children , 2008, Proceedings of the National Academy of Sciences.

[48]  Robinson G. Hollister,et al.  How Close Is Close Enough? Evaluating Propensity Score Matching Using Data From A Class Size Reduction Experiment , 2007 .

[49]  G. Duncan,et al.  Neighborhood resources, racial segregation, and economic mobility: Results from the Gautreaux program , 2006 .

[50]  Michael Howell-Moroney.  The Geography of Opportunity and Unemployment: An Integrated Model of Residential Segregation and Spatial Mismatch , 2005 .

[51]  Thomas W Sanchez,et al.  Race, space, and unemployment duration , 2005 .

[52]  Giorgio Topa,et al.  Place of Work and Place of Residence: Informal Hiring Networks and Labor Market Outcomes , 2005, Journal of Political Economy.

[53]  G. Duncan,et al.  Participation in a residential mobility program from the client's perspective: Findings from Gautreaux Two , 2005 .

[54]  J. Brooks-Gunn,et al.  Short-term effects of moving from public housing in poor to middle-class neighborhoods on low-income, minority adults' outcomes. , 2004, Social science & medicine.

[55]  Bruce A. Weinberg,et al.  Do Neighborhoods Affect Hours Worked? Evidence from Longitudinal Data , 2004, Journal of Labor Economics.

[56]  P. Murphy,et al.  NEIGHBORHOODS AND HEALTH , 2004 .

[57]  Virginia Parks,et al.  Access to Work: The Effects of Spatial and Social Accessibility on Unemployment for Native-Born Black and Immigrant Women in Los Angeles , 2004 .

[58]  Malcolm Maclure,et al.  Clinical and economic consequences of a reimbursement restriction of nebulised respiratory therapy in adults: direct comparison of randomised and observational evaluations , 2004, BMJ : British Medical Journal.

[59]  Howard S. Bloom,et al.  Can Propensity-Score Methods Match the Findings from a Random Assignment Evaluation of Mandatory Welfare-to-Work Programs? , 2004, Review of Economics and Statistics.

[60]  Roberto Agodini,et al.  Are Experiments the Only Option? A Look at Dropout Prevention Programs , 2004, Review of Economics and Statistics.

[61]  Philip Oreopoulos.  The Long-Run Consequences of Living in a Poor Neighborhood , 2003 .

[62]  Olga V. Demler,et al.  The epidemiology of major depressive disorder: results from the National Comorbidity Survey Replication (NCS-R). , 2003, JAMA.

[63]  B. Jacob.  Public Housing, Housing Vouchers and Student Achievement: Evidence from Public Housing Demolitions in Chicago , 2003 .

[64]  R. Kessler,et al.  The performance of the K6 and K10 screening scales for psychological distress in the Australian National Survey of Mental Health and Well-Being , 2003, Psychological Medicine.

[65]  E. Silver,et al.  Neighborhood structural characteristics and mental disorder: Faris and Dunham revisited. , 2002, Social science & medicine.

[66]  R. Sampson,et al.  ASSESSING "NEIGHBORHOOD EFFECTS": Social Processes and New Directions in Research , 2002 .

[67]  S. Danziger,et al.  Proximity and opportunity: How residence and race affect the employment of welfare recipients , 2002 .

[68]  C. Ross,et al.  Neighborhood disadvantage, disorder, and health. , 2001, Journal of health and social behavior.

[69]  David A. Reingold,et al.  Does Urban Public Housing Diminish the Social Capital and Labor Force Activity of Its Tenants , 2001 .

[70]  D. Phillips,et al.  From Neurons to Neighborhoods: The Science of Early Childhood Development , 2000 .

[71]  R. Margo,et al.  RESIDENTIAL SEGREGATION AND SOCIOECONOMIC OUTCOMES: WHEN DID GHETTOS GO BAD? , 2000 .

[72]  C. Ross,et al.  The Contingent Meaning of Neighborhood Stability for residents’ psychological well-being , 2000, American Sociological Review.

[73]  Leonard S. Rubinowitz,et al.  Crossing the Class and Color Lines: From Public Housing to White Suburbia , 2000 .

[74]  J. Brooks-Gunn,et al.  The neighborhoods they live in: the effects of neighborhood residence on child and adolescent outcomes. , 2000, Psychological bulletin.

[75]  G. Kaplan,et al.  Neighborhood social environment and risk of death: multilevel evidence from the Alameda County Study. , 1999, American journal of epidemiology.

[76]  James R. Elliott.  Social Isolation and Labor Market Insulation: Network and Neighborhood Effects on Less-Educated Urban Workers , 1999 .

[77]  G. Kaplan,et al.  Poverty area residence and changes in depression and perceived health status: evidence from the Alameda County Study. , 1999, International journal of epidemiology.

[78]  K. Smith,et al.  Phantom of the area: poverty-area residence and mortality in the United States. , 1998, American journal of public health.

[79]  Ronald W. Manderscheid,et al.  Neighborhood characteristics and mental illness , 1998 .

[80]  T. Vartanian.  Neighborhood Effects on AFDC Exits: Examining the Social Isolation, Relative Deprivation, and Epidemic Theories , 1997, Social Service Review.

[81]  S. Raudenbush,et al.  Neighborhoods and violent crime: a multilevel study of collective efficacy. , 1997, Science.

[82]  M. A. Thompson.  The Impact of Spatial Mismatch on Female Labor Force Participation , 1997 .

[83]  A. Ellaway,et al.  Does area of residence affect body size and shape? , 1997, International Journal of Obesity.

[84]  H. Hoynes.  Local Labor Markets and Welfare Spells: Do Demand Conditions Matter? , 1996, Review of Economics and Statistics.

[85]  William Julius Wilson,et al.  When Work Disappears: New Implications for Race and Urban Poverty in the Global Economy , 1998, Celebrating 40 Years of Ethnic and Racial Studies.

[86]  I. Luckey.  : American Apartheid: Segregation and the Making of the Underclass , 1995 .

[87]  E. Glaeser,et al.  Are Ghettos Good or Bad? , 1995 .

[88]  Harry J. Holzer,et al.  The Spatial Mismatch Hypothesis: What Has the Evidence Shown? , 1991 .

[89]  W. Wilson,et al.  The Truly Disadvantaged: The Inner City, The Underclass, and Public Policy. , 1988 .

[90]  R. Lalonde.  Evaluating the Econometric Evaluations of Training Programs with Experimental Data , 1984 .

[91]  Mark S. Granovetter.  The Strength of Weak Ties , 1973, American Journal of Sociology.

[92]  J. Kain.  Housing Segregation, Negro Employment, and Metropolitan Decentralization , 1968 .

[93]  Vivian C. Wong,et al.  Working Paper : Evaluating Methods for Selecting School-Level Comparisons in Quasi-Experimental Designs : Results from a Within-Study Comparison , 2016 .

[94]  Sean F. Reardon,et al.  Residential Segregation by Income , 1970-2009 , 2013 .

[95]  Lawrence F. Katz,et al.  What Can We Learn about Neighborhood Effects from the Moving to Opportunity Experiment? , 2009 .

[96]  William J. Smith,et al.  Does residential location matter to the employment of TANF recipients? Evidence from a dynamic discrete choice model with unobserved effects☆ , 2008 .

[97]  村上 あかね,et al.  パネル・リポート アメリカのパネル調査--Panel Study of Income Dynamics , 2008 .

[98]  Geography of Consumer Bankruptcies and Neighborhood Characteristics , 2006 .

[99]  D. McKenzie,et al.  How Important is Selection? Experimental vs. Non-Experimental Measures of the Income Gains from Migration , 2006, SSRN Electronic Journal.

[100]  K. O’Regan,et al.  Teenage Employment and the Spatial Isolation of Minority and Poverty Households , 2004 .

[101]  Robin Jacob,et al.  Moving to Opportunity for Fair Housing Demonstration Program , 2003 .

[102]  Laura Leete,et al.  Public Housing Assistance, Public Transportation, and the Welfare-to-Work Transition , 2003 .

[103]  Robin Jacob,et al.  Moving to Opportunity for Fair Housing Demonstration Program , 2003 .

[104]  A. Ellaway,et al.  Neighborhoods and health: overview , 2003 .

[105]  Todd A. Vermilyea.  Who is Unbanked , and Why : Results from a Large , New Survey of Low-and-Moderate Income Adults , 2002 .

[106]  A. D. Diez Roux,et al.  Investigating neighborhood and area effects on health. , 2001, American journal of public health.

[107]  Catherine E. Ross,et al.  Neighborhood disadvantage and adult depression , 2000 .

[108]  Elijah Anderson.  Code of the Street: Decency, Violence, and the Moral Life of the Inner City , 1999 .

[109]  Paul A. Jargowsky,et al.  Poverty and Place: Ghettos, Barrios, and the American City , 1998 .

[110]  James E. Rosenbaum,et al.  Changing the Geography of Opportunity by Expanding Residential Choice: Lessons from the Gautreaux Program , 1995 .

[111]  James E. Rosenbaum,et al.  Employment and earnings of low-income blacks who move to middle-class suburbs , 1991 .

[112]  Laurence E. Lynn,et al.  Inner-city poverty in the United States , 1990 .

[113]  Rebecca A. Maynard,et al.  The Adequacy of Comparison Group Designs for Evaluations of Employment-Related Programs , 1987 .