Developing measures to assess constructs from the Inner Setting domain of the Consolidated Framework for Implementation Research

BackgroundScientists and practitioners alike need reliable, valid measures of contextual factors that influence implementation. Yet, few existing measures demonstrate reliability or validity. To meet this need, we developed and assessed the psychometric properties of measures of several constructs within the Inner Setting domain of the Consolidated Framework for Implementation Research (CFIR).MethodsWe searched the literature for existing measures for the 7 Inner Setting domain constructs (Culture Overall, Culture Stress, Culture Effort, Implementation Climate, Learning Climate, Leadership Engagement, and Available Resources). We adapted items for the healthcare context, pilot-tested the adapted measures in 4 Federally Qualified Health Centers (FQHCs), and implemented the revised measures in 78 FQHCs in the 7 states (N = 327 respondents) with a focus on colorectal cancer (CRC) screening practices. To psychometrically assess our measures, we conducted confirmatory factor analysis models (CFA; structural validity), assessed inter-item consistency (reliability), computed scale correlations (discriminant validity), and calculated inter-rater reliability and agreement (organization-level construct reliability and validity).ResultsCFAs for most constructs exhibited good model fit (CFI > 0.90, TLI > 0.90, SRMR < 0.08, RMSEA < 0.08), with almost all factor loadings exceeding 0.40. Scale reliabilities ranged from good (0.7 ≤ α < 0.9) to excellent (α ≥ 0.9). Scale correlations fell below 0.90, indicating discriminant validity. Inter-rater reliability and agreement were sufficiently high to justify measuring constructs at the clinic-level.ConclusionsOur findings provide psychometric evidence in support of the CFIR Inner Setting measures. Our findings also suggest the Inner Setting measures from individuals can be aggregated to represent the clinic-level. Measurement of the Inner Setting constructs can be useful in better understanding and predicting implementation in FQHCs and can be used to identify targets of strategies to accelerate and enhance implementation efforts in FQHCs.

[1]  P. Greenbaum,et al.  Multilevel Confirmatory Factor Analysis of a Scale Measuring Interagency Collaboration of Children’s Mental Health Agencies , 2011, Journal of emotional and behavioral disorders.

[2]  P. Bentler,et al.  Comparative fit indexes in structural models. , 1990, Psychological bulletin.

[3]  James R McKay,et al.  Implementation of evidence-based substance use disorder continuing care interventions. , 2011, Psychology of addictive behaviors : journal of the Society of Psychologists in Addictive Behaviors.

[4]  P. Basch.  Quality of health care delivered to adults in the United States. , 2003, New England Journal of Medicine.

[5]  R. Palmer,et al.  Effect of Facilitation on Practice Outcomes in the National Demonstration Project Model of the Patient-Centered Medical Home , 2010, The Annals of Family Medicine.

[6]  Carlos Roberto Jaén,et al.  Methods for Evaluating Practice Change Toward a Patient-Centered Medical Home , 2010, The Annals of Family Medicine.

[7]  Albert Satorra,et al.  A scaled difference chi-square test statistic for moment structure analysis , 1999 .

[8]  Maria E Fernandez,et al.  Practice adaptive reserve and colorectal cancer screening best practices at community health center clinics in 7 states , 2015, Cancer.

[9]  Bryan J Weiner,et al.  The meaning and measurement of implementation climate , 2011, Implementation science : IS.

[10]  A. Kuniyuki,et al.  Capability for change at community health centers serving Asian Pacific Islanders: an exploratory study of a cancer screening evidence-based intervention. , 2013, Asian Pacific journal of cancer prevention : APJCP.

[11]  R. Beidas,et al.  It Takes a Village: A Mixed Method Analysis of Inner Setting Variables and Dialectical Behavior Therapy Implementation , 2015, Administration and Policy in Mental Health and Mental Health Services Research.

[12]  Bryan J Weiner,et al.  Conceptualization and measurement of organizational readiness for change: a review of the literature in health services research and other fields. , 2008, Medical care research and review : MCRR.

[13]  Timothy J. Robinson,et al.  Multilevel Analysis: Techniques and Applications , 2002 .

[14]  G. Robert,et al.  Diffusion of innovations in service organizations: systematic review and recommendations. , 2004, The Milbank quarterly.

[15]  E A Balas,et al.  Improving preventive care by prompting physicians. , 2000, Archives of internal medicine.

[16]  S. Kozlowski,et al.  Multilevel Theory, Research, and Methods in Organizations: Foundations, Extensions, and New Directions , 2000 .

[17]  Luke Wolfenden,et al.  Psychometric properties of implementation measures for public health and community settings and mapping of constructs against the Consolidated Framework for Implementation Research: a systematic review , 2016, Implementation Science.

[18]  C. Klabunde,et al.  Health plan policies and programs for colorectal cancer screening: a national profile. , 2004, The American journal of managed care.

[19]  G. A. Marcoulides,et al.  Multilevel Analysis Techniques and Applications , 2002 .

[20]  C. Helfrich,et al.  Implementation Science Organizational Readiness to Change Assessment (orca): Development of an Instrument Based on the Promoting Action on Research in Health Services (parihs) Framework , 2022 .

[21]  Elisa M. Torres,et al.  Validating the Implementation Climate Scale (ICS) in child welfare organizations. , 2016, Child abuse & neglect.

[22]  Peggy A. Hannon,et al.  The Cancer Prevention and Control Research Network: An Interactive Systems Approach to Advancing Cancer Control Implementation Research and Practice , 2014, Cancer Epidemiology, Biomarkers & Prevention.

[23]  C. Lance,et al.  The Sources of Four Commonly Reported Cutoff Criteria , 2006 .

[24]  Michelle Kegler,et al.  Measuring constructs from the Consolidated Framework for Implementation Research in the context of increasing colorectal cancer screening at community health centers , 2015, Implementation Science.

[25]  J. Dawson,et al.  Validating the organizational climate measure: links to managerial practices, productivity and innovation , 2005 .

[26]  K. Stange,et al.  Journey to the Patient-Centered Medical Home: A Qualitative Analysis of the Experiences of Practices in the National Demonstration Project , 2010, The Annals of Family Medicine.

[27]  Robert F. DeVellis,et al.  Scale Development: Theory and Applications. , 1992 .

[28]  M. Slaughter,et al.  An intervention to improve program implementation: findings from a two-year cluster randomized trial of Assets-Getting To Outcomes , 2013, Implementation Science.

[29]  K. Ribisl,et al.  Impact of the Cancer Prevention and Control Research Network: Accelerating the Translation of Research Into Practice. , 2017, American journal of preventive medicine.

[30]  P. Bentler,et al.  Cutoff criteria for fit indexes in covariance structure analysis : Conventional criteria versus new alternatives , 1999 .

[31]  Stephanie L. Castro,et al.  Data analytic methods for the analysis of multilevel questions: A comparison of intraclass correlation coefficients, rwg(j), hierarchical linear modeling, within- and between-analysis, and random group resampling , 2002 .

[32]  Cara C. Lewis,et al.  Outcomes for implementation science: an enhanced systematic review of instruments using evidence-based rating criteria , 2015, Implementation Science.

[33]  Kimberly E. Hoagwood,et al.  Measures for Predictors of Innovation Adoption , 2015, Administration and Policy in Mental Health and Mental Health Services Research.

[34]  B. Muthén,et al.  Multilevel Covariance Structure Analysis , 1994 .

[35]  Amy Buhl Conn,et al.  Implementing computerized technology: an organizational analysis. , 2001, The Journal of applied psychology.

[36]  Phil Wood.  Confirmatory Factor Analysis for Applied Research , 2008 .

[37]  Lisa M Klesges,et al.  The future of health behavior change research: What is needed to improve translation of research into health promotion practice? , 2004, Annals of behavioral medicine : a publication of the Society of Behavioral Medicine.

[38]  J. Lowery,et al.  Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science , 2009, Implementation science : IS.

[39]  K. Roback,et al.  Factors influencing rheumatologists’ prescription of biological treatment in rheumatoid arthritis: an interview study , 2014, Implementation Science.

[40]  Maria E Fernandez,et al.  The Cancer Prevention and Control Research Network , 2004, Preventing chronic disease.

[41]  Shin-Ping Tu,et al.  Systems Antecedents for Dissemination and Implementation , 2012, Health education & behavior : the official publication of the Society for Public Health Education.

[42]  D. Simpson,et al.  Assessing organizational readiness for change. , 2002, Journal of substance abuse treatment.