 |
| Photo courtesy Dr. Bottema-Beutel
[image: Formal photo of Dr. Bottema-Beutel, a smiling white woman with medium-length side parted brown hair.] |
Advocates of early autism interventions often claim such approaches are “evidence based,” whereas critics have long pointed out individual flaws in cited studies.
We were glad to learn about Dr. Kristen Bottema-Beutel’s analysis of general conflicts of interest in early autism research, and talk with her about how her findings complicate assertions about being early autism interventions being evidence based, and what else she and her team discovered.
—-
Thinking Person’s Guide to Autism: Can you tell us why you decided to pursue this analysis of conflicts of interest (COIs) in early autism intervention research?
Bottema-Beutel: The short answer is that I’ve been following Michelle Dawson on Twitter (her handle is @autismcrisis). Michelle is an autistic researcher who has been sounding the alarm on undisclosed COIs for more than a decade—before I even entered graduate school. Michelle gives really insightful critical commentary on a huge amount of autism research, and regularly tweets about intervention studies that do not disclose when the researchers could potentially have something to gain from positive findings.
Once you start looking, it’s hard not to notice that a very large portion of intervention research is conducted by the same people who designed the interventions, or who provide the intervention. This isn’t necessarily a problem, but you do at some point want independent replications, which means that researchers not associated with developing or providing the intervention conduct studies to determine how well they work. There are many decisions that go into designing, implementing, and interpreting intervention studies, and researchers might not always be aware of making decisions that ‘bias’ the study—which means they show positive results even if the intervention doesn’t have a positive effect.
A few years ago, my colleagues Micheal Sandbank (UT Austin), Tiffany Woynaroski (Vanderbilt) and I decided to do what’s called a ‘meta-analysis’, which involved searching for all the studies we could find on interventions for young children with autism, and combining all the data together to see if we could statistically determine which types of interventions have positive effects. These results were published a few months prior to the COI study.
While we were doing this meta-analysis and had all of the literature together in one place, it seemed like a good opportunity to systematically explore what Michelle had been noticing and pointing out for a long time. My graduate student Shannon Crowley worked on both of these studies, and was instrumental in coding COIs and interpreting our results.
TPGA: Proponents of early autism interventions often describe them as “evidence-based.” Does your analysis complicate these claims?
Bottema-Beutel: I think it does. One of our findings from the meta-analysis is that we haven’t conducted enough high-quality studies on any given intervention to make bold claims about what works. The term ‘evidence-based’ is a bit arbitrary, and different research groups have different quality standards they use when categorizing an intervention as evidence-based.
Another perspective I gained from following Michelle Dawson is that conducting high-quality research is a social justice issue. Autistic people deserve research that adheres to quality standards that are agreed upon by researchers across disciplines—we shouldn’t have a separate set of lower standards that we only apply to autism research.
Studies should also be conducted by researchers who are transparent about their COIs, and how these COIs could potentially bias results. Having a COI does not automatically mean that your research is biased, but properly disclosing a COI invites additional scrutiny of your methods and interpretations, which is a good thing. We found research from other areas in psychology where COIs were fully disclosed, and the researchers provided very detailed descriptions of how they built in safeguards in their study designs to ensure that the COIs didn’t compromise the study findings. We could do this in autism research.
One thing I think I should add here is that not all of the responsibility for COI under-reporting falls on researchers. Part of the problem is that the journals where these studies are submitted have rather vague policies about which COIs should be disclosed. We’re hoping that with our paper, researchers will have a bit more information about what constitutes a COI, journal editors will improve the way they communicate COI disclosure policies to researchers, and the entire submission process will involve more vigilance to ensure researchers disclose COIs properly.
As I mentioned before, the presence of a COI doesn’t necessarily mean the study is biased. But, if COIs are more readily disclosed, we could determine which kind of COIs are associated with researcher bias, and researchers could then take steps to ensure that those biases don’t influence their study.
TPGA: You examined eight types of COIs. What were your criteria for choosing them?
Bottema-Beutel: We chose these eight because they were examined in similar studies in other fields, and because they corresponded with the types of COIs that researchers studying interventions geared towards young children with autism might be likely to have.
TPGA: Were there other COIs you didn’t include, and if so, why?
Bottema-Beutel: We didn’t actively exclude any COI categories that we were able to find. One COI that we have a feeling is pretty prevalent but that we were unable to find much evidence for is the acceptance of speaker fees. It’s pretty common for researchers to be asked to present their work, especially if they publish research showing promising intervention results. Sometimes, these invitations come with stipends to compensate researchers for their time.
We found only one study that disclosed an author had received speaker fees, and it turned out the speaker fees were for speaking on a topic unrelated to the intervention being studied. There’s no real internet record of researchers receiving speaker fees (unlike many of the other COIs we examined), so we were unable to get good data for this.
TPGA: Your analysis goes back to 1970. Did you notice any trends, such as decreasing or increasing COIs over time?
Bottema-Beutel: This is a really great question! I wish we could do an analysis like this, but because most of our data is comprised of undisclosed COIs that we located through internet searches, we probably would see that all of the COIs we tracked increased over time, but only because they were better documented on the web as researchers’ web presences increased over time.
TPGA: Do issues with COIs correspond with which type of intervention (ABA, behavioral, etc) is being studied?
Bottema-Beutel: Newer intervention approaches, like technology-based and “naturalistic-developmental-behavioral-interventions” (a combination of approaches that was formally defined as an approach just a few years ago) have a relatively higher number of COIs related to the developer of the intervention also being the researcher. In addition to newer studies just having COIs that are easier to find, this is likely because a second generation of researchers hasn’t been in the field long enough to secure the big grant funding required for an independent evaluation of these approaches.
One issue preventing a complete answer for this question is that our study only examined “group design” studies, and not all intervention approaches tend to be examined with group design studies. For example, the vast majority of the ABA literature uses a different approach, called single-case design. We didn’t look at any of this research, but if we had, we might find many more COIs in this group of studies.
In an online conversation, an ABA practitioner and researcher pointed out that many behavioral researchers are also practicing clinicians, and that they are not generally aware that this dual role inherently constitutes a COI. This is a big problem in my opinion, and means that researchers need to be better trained about this topic.
In the group design research that we did review, all of the different intervention types we examined were associated with at least some COIs, except cognitive behavior therapy—but this approach only had a single study involving young children with autism.
TPGA: You included a list of studies analyzed in the supporting information—do you intend to also release the coding of undetected COIs for each study at some point or make that information available for secondary analysis?
Bottema-Beutel: If any researchers wanted this information for secondary analysis (or to simply check our work) we would be happy to share it with them.
TPGA: What do you consider the takeaways from your analysis, especially for families considering early autism interventions?
Bottema-Beutel: Our take home message for families is that they should continue to use intervention services that are working for them. However, families should also be informed about what a conflict of interest is, and should consider whether they are being encouraged to use a particular intervention by someone who has a conflict of interest.
Our message for researchers and funders is that they should prioritize conducting high-quality, independent replications of the interventions that we currently think show promising evidence of effectiveness. I know of some intervention proponents who claim that a particular intervention approach has the “best” evidence base simply because they have more studies, but volume alone isn’t convincing to me. We need to look at the quality of the evidence, and the extent to which we think the evidence might be biased by COIs.
Kristen Bottema-Beutel is an Associate Professor in the Teaching, Curriculum, and Society department in the Lynch School of Education and Human Development at Boston College; her research interests are in social interaction and development in autistic children and youth.
