Evaluating Bang for the Buck

Evaluators often use qualitative research methods, yet there is little evidence on the comparative cost-effectiveness of the two most commonly employed qualitative methods—in-depth interviews (IDIs) and focus groups (FGs). We performed an inductive thematic analysis of data from 40 IDIs and 40 FGs on the health-seeking behaviors of African American men (N = 350) in Durham, North Carolina. We used a bootstrap simulation to generate 10,000 random samples from each data set and calculated the number of data collection events necessary to reach different levels of thematic saturation. The median number of data collection events required to reach 80% and 90% saturation was 8 and 16, respectively, for IDIs and 3 and 5 for FGs. Interviews took longer but were more cost-effective at both levels. At the median, IDIs cost 20–36% less to reach thematic saturation. Evaluators can consider these empirically based cost-effectiveness data when selecting a qualitative data collection method.

[1]  G. Guest,et al.  How Many Focus Groups Are Enough? Building an Evidence Base for Nonprobability Sample Sizes , 2017 .

[2]  M. Sandelowski.  Sample size in qualitative research. , 1995, Research in nursing & health.

[3]  J. Knodel,et al.  The Design and Analysis of Focus Group Studies: A Practical Approach , 1993 .

[4]  Eleanor McLellan,et al.  Beyond the Qualitative Interview: Data Preparation and Transcription , 2003 .

[5]  J. Morse.  The Significance of Saturation , 1995 .

[6]  Christina A. Christie,et al.  Insight Into Evaluation Practice: A Content Analysis of Designs and Methods Used in Evaluation Studies Published in North American Evaluation-Focused Journals , 2010 .

[7]  D. Stokes,et al.  Methodology or “methodolatry”? An evaluation of focus groups and depth interviews , 2006 .

[8]  Alarcos Cieza,et al.  Individual interviews and focus groups in patients with rheumatoid arthritis: a comparison of two qualitative methods , 2011, Quality of Life Research.

[9]  James C McDaid,et al.  Program Evaluation and Performance Measurement: An Introduction to Practice , 2006 .

[10]  Brad Wardman,et al.  Voice of the customer , 2013, 2013 APWG eCrime Researchers Summit.

[11]  Edward F. Fern.  The use of Focus Groups for Idea Generation: The Effects of Group Size, Acquaintanceship, and Moderator on Response Quantity and Quality , 1982 .

[12]  F. Guillemin,et al.  Content of quality-of-life instruments is affected by item-generation methods. , 2007, International journal for quality in health care : journal of the International Society for Quality in Health Care.

[13]  Greg Guest,et al.  Collecting Qualitative Data: A Field Manual for Applied Research , 2012 .

[14]  Patrick J. O'Connor,et al.  Selecting Individual or Group Interviews , 1993 .

[15]  Janet Mancini Billson,et al.  Focus Groups: A Practical Guide for Applied Research , 1989 .

[16]  D. Morgan.  The Focus Group Guidebook , 1997 .

[17]  Kathleen M. MacQueen,et al.  Applied Thematic Analysis , 2011 .

[18]  J. Ivey.  Focus groups. , 2011, Pediatric nursing.

[19]  Peter Kuster,et al.  Handbook Of Practical Program Evaluation , 2016 .