By

Selection bias is one of the biggest pitfalls in research into psychedelics. Not because researchers are being “dishonest,” but because in practice it is difficult to recruit a truly representative group of participants. A recent scientific article clearly demonstrates this: people from an enthusiastic user group report significantly more benefits from psychedelic experiences than people from a more general sample. This does not automatically mean that psychedelics “don’t work,” but it does mean that we must be cautious about drawing major conclusions based on self-selecting participants.

In this article, we explain what selection bias is, why it is particularly relevant to psychedelics, exactly what the researchers found, and how to interpret these types of results wisely. We make a clear distinction between research, self-reporting, and practical experience. For those who wish to read the source: the article discussed is summarized at Trip forum.

What is selection bias and why does it matter?

Selection bias arises when the people included in a study are systematically different from the people about whom you actually want to make statements. This can lead to outcomes appearing more positive or negative than they would be in a broader population.

Selection bias is to be expected with psychedelics. Many participants sign up because they are curious, have heard positive stories, have had good experiences before, or view psychedelics as a means for personal growth. This is a different starting point than someone who is skeptical, fearful of losing control, or has had a difficult experience previously. If you primarily measure the first group, you can also expect positive effects more frequently in the outcomes.

This is not a detail. It touches directly on the question: are we measuring an effect of the substance, or are we primarily measuring an effect of who participates and under what circumstances?

What exactly did this article investigate?

The researchers wanted to investigate whether the reported benefit of psychedelic experiences differs between:

1) a “convenience sample” of psychedelics enthusiasts (people from a context where interest is already high), and

2) a more general sample, recruited via Prolific, a platform often used to recruit more diverse groups of participants.

In total, 1,182 people who had previously used psychedelics participated. They completed an online questionnaire about, among other things:

• experienced impact on quality of life

• mindset (set) and setting around the experience

• motivation for use (for example, personal growth)

• personality traits (Big Five)

Important: this study was not about a clinical treatment or a guided session, but about the interpretation of research outcomes and how recruitment can influence the results.

The core finding: enthusiasts report much more benefit

The outcome was clear. The enthusiastic user group reported a much more positive impact on quality of life than the more general sample. The difference was large, with an effect size of d = 0.84. In the social and behavioral sciences, this is generally considered substantial.

In addition, the groups also differed on factors that are often associated with a more positive experience:

• Enthusiasts scored higher on openness, extraversion, and agreeableness.

• They reported a more favorable mindset and setting.

• They cited personal growth as a motive more often.

In other words: the group that you would expect beforehand to be “more receptive” to positive experiences indeed reported more positive impact.

Why statistical correction does not solve everything

The researchers statistically corrected for mindset, setting, motivation, and personality. You would think: if you correct for these, the difference between the groups would disappear. But that did not happen. Even after correction, the group difference persisted.

In the analysis used (ANCOVA), “group membership” actually proved to be the strongest predictor of the reported impact on quality of life, followed by setting, motivation, openness, and mindset.

That does not automatically mean that there is a “mystery variable”. It could also mean that selection bias is at play at multiple levels, for example through:

• expectations that are not fully captured in a questionnaire

• differences in recall and interpretation of the experience

• community influence and language to interpret experiences

• a greater willingness to attribute positive effects to psychedelics

An important insight is that correcting for a few measurable factors is not the same as “erasing” selection bias. Certainly not with complex, subjective experiences.

What this research does not demonstrate

Nuance is essential. This was a cross-sectional online survey based on self-reporting. That has limitations:

• It does not demonstrate a causal link. You cannot say: psychedelics cause this improvement.

• It does not demonstrate objective health improvement. It concerns perceived impact, not clinical measurements.

• It is susceptible to recall bias (how people look back) and to interpretive frameworks (how people assign meaning).

The article therefore does not say that psychedelics can have no value. It does say, however: if you primarily recruit enthusiastic participants, the outcomes will likely systematically appear more positive than when you survey a broader group.

What does this mean for how we talk about psychedelics?

The conversation about psychedelics often precedes the evidence. There are promising lines of research, but also many open questions. This article is primarily a call to keep language and conclusions precise.

For readers, it means, for example:

• Be critical of major success stories, especially if they come from communities with high expectations.

• Look at how participants were recruited. That often partly determines the outcome.

• Distinguish between “people experience it as helpful” and “it has been proven effective as a treatment”. These are different claims.

For researchers, it means: invest in more representative samples and be transparent about recruitment. For practitioners, it means: discuss expectations and context, because set and setting in particular are strongly correlated with reported impact.

And what about MDMA and the therapy context?

Strictly speaking, MDMA does not fall under the category of classic psychedelics, but in public debate, the themes are often discussed together: expectations, setting, guidance, and the tendency toward positive self-selection. The same mechanism may be at play there as well: people who feel drawn to a session may already have more hope, motivation, or preparation than someone who would not participate.

It is important to remain factual: MDMA sessions can currently only be discussed within scientific research or in clinical practice via harm reduction. It is therefore particularly relevant to handle expectations, screening, preparation, and aftercare with care, without making medical claims or guarantees.

Anyone wishing to get a general overview of how an MDMA session is approached in a harm-reduction context can read information and possibly request an introductory meeting via sign up for MDMA session. That is not a medical treatment and not a promise of results, but a way to explore what is and is not possible within safe, realistic frameworks.

Conclusion

This study convincingly demonstrates that selection bias can have a major influence on how positive the benefits of psychedelics appear in questionnaire-based research. Enthusiasts report much greater gains in quality of life than a more general sample, and that difference persists even after correcting for several known factors. The most important lesson is not that psychedelics “do” or “don’t” work, but that recruitment, expectations, and context all contribute to determining which outcomes are measured. A fairer picture requires more representative samples and careful interpretation of self-reports.