Evaluating the Quality of the Evidence

Core Concepts

Study design and the evidence hierarchy

For questions about whether an intervention works, study designs form a rough hierarchy from strongest to weakest:

1. Systematic reviews and meta-analyses — transparent syntheses of multiple studies.
2. Randomized controlled trials (RCTs) — participants randomly assigned to intervention or control; randomization balances known and unknown differences.
3. Quasi-experimental studies — an intervention compared with a comparison group without random assignment (common in community settings).
4. Cohort studies — groups defined by exposure are followed forward in time.
5. Case-control studies — people with and without an outcome are compared looking backward.
6. Cross-sectional studies — a snapshot of exposure and outcome at one time.
7. Case reports, expert opinion — single cases or informed views, useful for hypotheses, not proof.

The hierarchy is a starting point, not a verdict: a poorly done RCT can be less trustworthy than a well-done quasi-experiment. Public health often relies on quasi-experimental and natural experiments because randomizing whole communities is often impossible or unethical — appraise conduct, not label.

Internal validity: is the finding true?

Internal validity asks whether the study's results reflect reality for the people studied. Key questions:

• Was there a comparison? Without one, improvement cannot be attributed to the intervention — the community may have improved anyway.
• Were groups comparable? Randomization helps; otherwise check that groups were similar on age, income, health, and other factors that could explain the result.
• Was follow-up complete? High dropout makes results unrepresentative.
• Were outcomes measured the same way for everyone? Different measurement across groups (information bias) distorts results.
• Could confounding explain it? A third factor tied to both exposure and outcome may be the real driver.

Bias and confounding

• Selection bias: who enters or stays in the study differs systematically between groups (e.g., healthier, more motivated people join the program).
• Information (measurement) bias: outcomes or exposures are recorded inaccurately or differently between groups.
• Confounding: a third variable is tangled with the exposure and independently affects the outcome. People who join an exercise program may also eat better — diet confounds the evaluation of exercise alone. Randomization is the strongest guard; statistical adjustment is only a partial fix.

External validity: will it work here?

A valid study may still not apply to your community. Ask:

• Are the people similar? Age, language, culture, income, and baseline health matter.
• Is the setting similar? Urban versus rural, school versus clinic, high-resource versus low-resource.
• Can we deliver the same dose and quality? Staffing, training, and fidelity affect results.
• Is it feasible and acceptable here? The community may not want or be able to sustain the program.

Evidence transfers with judgment, not automatically.

Effect size, precision, and significance

• Effect size is how big the difference is — a program can be statistically detectable yet trivially small in practice.
• Precision is shown by confidence intervals: a wide interval means the true effect is poorly pinned down; a narrow one means the estimate is stable.
• Statistical significance says the result is unlikely to be due to chance; public health significance asks whether the effect is big enough, in enough people, to matter. Both are needed.

Appraisal tools and reporting standards

You need not reinvent appraisal — checklists and standards exist:

• CASP checklists — free, question-by-question appraisal guides for each study type.
• GRADE — a system for rating the certainty of a body of evidence and the strength of guideline recommendations.
• Reporting standards — CONSORT (trials), STROBE (observational studies), PRISMA (reviews) — make it easy to see what a study did; one that omits key details cannot be fully trusted.
• Conflicts of interest: check funding and author disclosures — conflicts can bias even otherwise careful studies.

Common Confusions