Researcher Requests for Inappropriate Analysis and Reporting: A U.S. Survey of Consulting Biostatisticians

The published literature on research misconduct, including a 2016 Cochrane report, has been evolving slowly, with notable growth during the past decade (112). These publications have addressed a wide array of topics, such as the prevalence of and reasons for article retraction (1317), types of misrepresentation of scientific findings (18), recognition of the complexity of research integrity in community and field research (1921), and calls for more and better training (2224). We regard inappropriate analysis and reporting of data as a form of research misconduct if their intent is to mislead those who use the research findings. Whether done intentionally or because of a lack of knowledge, inappropriate analysis and reporting of biomedical research remain a problem, despite advances in statistical methods and efforts to better educate researchers. This study aimed to quantify and describe requests for inappropriate analysis and reporting that biostatisticians receive from investigators during their biostatistical consultations. Methods Survey Instrument This U.S. national survey used the Bioethical Issues in Biostatistical Consulting (BIBC) Questionnaire, which was developed previously (25) (for the survey instrument, see the Supplement). The survey was pretested in a pilot study in which it was administered to a randomly drawn sample of 112 biostatisticians who were members of the American Statistical Association (ASA) (26). Supplement. Bioethics Survey in Data Analysis Survey items asked respondents how many times (using a 5-point scale: 0, 1, 2 to 4, 5 to 9, or 10) they had received specific requests from investigators for 18 inappropriate analysis and reporting practices during their biostatistical consultations. Respondents also were asked to rate the bioethical violation severity of each of the 18 potential inappropriate practices (on a 6-point scale ranging from least [0] to most [5] severe). Participants Our goal was to obtain completed surveys from 400 biostatisticians, and we encouraged their response with an ASA endorsement, the offer of a $99 Amazon gift certificate for survey completion, the use of an online data collection system that allowed respondents to remain anonymous, and instructions to report only requests received rather than actual inappropriate behavior. The ASA's Office of the Executive Director provided the research team with a database of 4000 persons randomly selected from the association's approximately 18000 registered members. For each member, the database included the membership ID number, name, e-mail address, affiliation, sex, race/ethnicity, birth year, highest degree achieved, employment category, and areas of specialty. The names were removed from the sample, and we retained those whose areas of specialty were listed as statistics, biostatistics, data analysis, or biometricsthat is, we removed those whose areas of specialty were not involved primarily in biostatistical consulting and data analysis and hence were inappropriate for our survey. This process resulted in a final database of 3874 members. Next, we generated a random number for each ASA record and sorted the numbers of the records included. We used Qualtrics survey software to distribute batches of 50 e-mail invitations at 2-day intervals and e-mailed reminders to nonresponders after 5 days. When we reached our target of 400 completed questionnaires, we ended the survey. These steps were taken to prevent overenrollment, which would have exceeded our budget for the incentives offered to respondents for their participation. By the time 400 responses were received, the Qualtrics e-mail records indicated that 800 e-mails had been distributed and 522 had been received and opened by the respondents. Statistical Analysis Statistical analysis of the data included descriptive analyses of the findings from the 18 bioethical violation questions as well as bivariate analyses using Pearson 2 tests for the 18 questions by the demographic variables of sex, age group, racial/ethnic group, and research tier of the university with which the respondent was affiliated. The bivariate analyses were performed with SPSS, version 24 (IBM). To determine how well our survey sample represented the overall population of ASA members, we conducted a statistical comparison on 4 demographic variables (sex, race/ethnicity, age, and highest degree achieved) between the sample and the overall ASA membership by using the Bonferroni criterion for multiple comparisons. Ethical Review The institutional review boards at the University of Maryland School of Public Health and New York University approved the study with expedited review because of minimal risk to the participants. Role of the Funding Source The Office of Research Integrity of the U.S. Department of Health and Human Services funded the study but had no role in the design, conduct, or analysis of the study; manuscript preparation; or decision to submit the manuscript for publication. Results Of 400 responses, 10 records were excluded because of a high level of incomplete data. Thus, our final sample of 390 represented a completion rate of 74.7% among members who received and opened our e-mail invitation (that is, the members successfully contacted), whereas the response rate for the survey was 48.8% for the 800 e-mails distributed (that is, regardless of whether the members received or opened the message) (26). Respondents mostly identified as male (64.1% men, 35.9% women); fell into 3 age groups: 23 to 39 (27.9%), 40 to 59 (40.1%), and 60 to 88 years (31.2%); and were predominately white (63.6% white, 23.8% Asian, and 12.6% other). Mean number of years working as a biostatistician was (SD, 13.0). No statistically significant differences were seen in sex, age, race/ethnicity, or highest degree obtained in a comparison between respondents and the 4000 members randomly selected by the ASA. Table 1 shows the reported severity and frequency of the 18 inappropriate requests in descending order by percentage of respondents who ranked that item as high severity (4 or 5 on a scale of 0 to 5). The proportion of respondents who rated a request as high severity ranged from 84% for requests to falsify statistical significance (such as the P value) to support a desired result to 33% for a request to not show a plot because it did not show an effect as strong as the researcher wanted. Table 1. Biostatistician-Reported Frequency and Severity Rating of Requests for Inappropriate Analysis and Reporting (n= 390)* Table 2 shows the comparisons of the reported frequency of the top 8 inappropriate requests (rated as high severity with frequent occurrence) by specific questions in the demographic section of the BIBC Questionnaire regarding age, race/ethnicity, and type of research university with which the biostatistician was affiliated. These requests are listed in the table (in an abbreviated format) according to the sequence asked on the BIBC Questionnaire. Of the 8 inappropriate requests, 6 demonstrated statistically significant differences by age, with younger biostatisticians more likely than their older colleagues to report a higher frequency. That 4 of the 8 requests for inappropriate analyses and reporting were statistically reported more often by respondents who identified as Asian or other than those who identified as white may be explained, in part, by age differences. Respondents who reported Asian or other race/ethnicity were younger than white respondents (mean age: Asian, 45.5 years; other, 46.7 years; white, 51.2 years). Among participants who worked in a first-tier research university (that is, one of 115 institutions the Carnegie Classification of Institutions of Higher Education categorized in 2018 as R1: Doctoral UniversitiesHighest Research Activity [27]), only 1 of the 8 requests reported differed in frequency from those made to respondents who did not work at such an institution. As the last row in Table 2 shows, the percentage of any age, racial/ethnic, or institution-type subgroup that reported receiving any of the 8 highest-severity requests for inappropriate analysis or reporting ranged from 54% to 84%. Table 2. Percentage of Respondents Reporting a Top 8 Violation Request, by Age, Race/Ethnicity, and Affiliation With a First-Tier Research University* Finally, we examined the missing responses for each of the 18 frequency and severity questions. For the 18 frequency items, 11 (mean, 0.6; range, 0 to 3) responses were missing, and for the 18 severity questions, 56 (mean, 3.1; range, 0 to 8) responses were missing. Across all 36 BIBC items with a possible 14040 responses (36 items390 respondents), the 67 missing responses represented only 0.005% of the total. Discussion This U.S. national survey of consulting biostatisticians suggests that requests by researchers for inappropriate analysis and reporting occur frequently. Of note, however, the 2 request types rated as highest severity (falsification of statistical significance and changing the data to achieve the desired outcome) were reported as the least frequently requested. Nonetheless, the frequency of inappropriate requests and the rating of many as high severity by respondents indicate a need to better educate researchers about the inappropriateness of such requests, which may represent poor-quality science at best and research misconduct at worst. The reasonably high survey response rate and the low rate of missing items bound the possible bias from nonresponse. Before the pilot study of our survey (28), the only published report we could identify that quantified researcher requests for inappropriate analysis and reporting was a 1998 international survey of biostatisticians who were members of the International Society for Clinical Biostatistics (29). Although the response rate was only 37%, the authors felt that the high proportion of respondents knowing about fraudulent projects [51%] provided the primary motivation for [publis