NEUTRON
    All articles
    Research
    Data Science

    Did it work, or did it just happen next? A study-design check for AI research

    Neutron Says Team5 min read

    A library adds text reminders and sees more people attend its workshops. An AI summary says the reminders increased attendance. The numbers may be right. The verb may not be.

    To say that something caused a result, you need a credible way to estimate what would have happened without it. You cannot observe the same people receiving and not receiving the same reminder at the same time. A well-designed comparison gives you a defensible substitute. A before-and-after chart, by itself, doesn’t.

    Here’s how to check what a study can support before you reuse it in a brief.

    1. Write down the claim you actually need

    Specify the intervention, group, outcome and period. “Do reminders work?” could mean more bookings, fewer no-shows, or more repeat visits.

    For this entirely fictional example, the question is: Did sending one text reminder the day before a library workshop increase attendance among registered adults during the spring pilot?

    That’s not a claim about library membership, other branches or different kinds of reminders. It’s specific enough to check against what the study actually measured.

    2. Find the comparison, not just the result

    Suppose the library reports that attendance rose from 60% in February to 72% in April after it introduced reminders. April also had Saturday workshops, while February’s sessions were on weeknights. In this example, attendance rose by 12 percentage points, but the effect of reminders is unknown. The schedule change is another plausible explanation, and the report doesn’t show how much either mattered.

    Compare three designs that might appear under the same headline:

    • Before and after, one group: February versus April at the same library. Useful for seeing that attendance changed. It doesn’t separate the reminders from other changes over time.
    • Two groups, no random assignment: One branch adopts reminders, another doesn’t. Check why each branch made that choice and whether attendance patterns, schedules and other outreach were comparable beforehand. A carefully designed nonrandomized study can support a causal estimate, but its comparison and assumptions need to hold up.
    • Random assignment: Eligible registrants are randomly assigned to receive the reminder or the usual confirmation. If the groups are followed and measured consistently, this is a stronger basis for estimating the effect of being assigned a reminder in that pilot.

    The U.S. Department of Education’s What Works Clearinghouse guide to quasi-experiments explains why baseline comparability matters and why one group’s before-and-after results do not provide an adequate comparison under its group design standards.

    3. Check what could still explain the difference

    Read the methods and results for answers to four questions:

    1. Who got what, and why? Random invitation to take a survey is not the same as random assignment to receive a service. If people chose their own group, ask what about that choice might also predict the outcome.
    2. Did anything else change? If the reminder group also gets extra staff follow-up, the study may estimate the effect of that package, not the text alone. If hours or location changed, note which groups were affected.
    3. Who is missing from the result? Check how many people were assigned and how many were included in the analysis. If missing attendance records or delivery failures were handled differently across groups, the original assignment may not protect the reported estimate.
    4. How was the outcome measured? Check that both groups used the same definition and data collection. A QR scan at one branch and a volunteer head count at another can produce differences that reflect measurement rather than attendance.

    The Cochrane Handbook’s chapter on bias in nonrandomized studies gives a deeper treatment of confounding, selection and measurement. Its context is intervention reviews; the same questions are useful for reading a program evaluation.

    4. Use a verb the design can support

    For the fictional before-and-after report, write:

    Attendance rose from 60% to 72% after reminders were introduced. The workshop schedule changed at the same time, and the report does not isolate the effect of reminders.

    If the report instead compared people who chose reminders with people who declined them, write “reminder recipients had higher attendance”, not “reminders increased attendance,” unless the study’s design and analysis justify that causal claim. For a credible experiment, report the estimated effect, the uncertainty the study provides, and who was tested. Even a good pilot does not prove the same effect will carry over to other branches.

    A fast rule: “after” describes timing; “associated with” describes a relationship; “caused” requires a design that supports that inference. Don’t grade a study by the strongest verb in its press release.

    Try this on a public study

    Use this prompt with one public report and its methods appendix. Then check the cited passages yourself.

    Evaluate the claim that [intervention] changed [outcome]. Identify how people or sites entered each group, what the comparison group experienced, any other changes during the study, missing outcomes and how outcomes were measured. Cite the relevant page or section for each answer. Write “not reported” where information is absent. Give me one sentence I can responsibly use and explain why any stronger wording would overstate the evidence.

    Try Neutron for a first pass on the public report. Keep the final claim tied to what the study tested, not what you hoped it would show.

    Share on social

    Related Articles