MUMultiple speciesBmoderateT0not transferable
Evidence levels grade confidence, not truth: what randomised, observational and animal studies support, how GRADE rates them and where the pyramid breaks.
Executive summary
An evidence level does not judge whether a statement is true. It judges how much confidence the available data deserve. The rating is given per outcome, not per study: in the same review, the cure rate can count as moderately certain while bone density counts as very uncertain. What matters is therefore not the label on the study design, but which alternative explanation that design can actually rule out.
Key points
An evidence level is a statement about the state of knowledge, not about nature. GRADE, the system adopted by numerous professional bodies worldwide, distinguishes four levels: high, moderate, low and very low certainty. These levels answer a single question: how likely is it that further good studies will still shift today’s result? A consensus article by an international methods group set out this framework in its current form in the British Medical Journal in 2008. It separates two things that were previously mixed up on a regular basis: the certainty of the evidence and the strength of any recommendation derived from it.
The most important point, and the one most often missed: certainty is assigned per outcome, not per study and not per article. A 2023 Cochrane review of parathyroid surgery pooled eight randomised trials with 447 adults. For the outcome of cure, the authors rated certainty as moderate; for bone density at the lumbar spine, in the same piece of work, as very low. Anyone who reads only the title and the conclusion will take both to be equally well supported.
Because every downgrade calls for a judgement. How large does an imprecision have to be before it counts? Is a contradiction between two studies real, or is it down to chance? GRADE makes these judgements visible and requires them to be justified, but it does not make them for anyone. Two careful teams can therefore reach different levels from the same body of data.
The classic ranking is usually drawn as a pyramid: case reports and laboratory experiments at the bottom, observational studies above them, then randomised trials, and systematic reviews at the top. This order is not a ranking of importance. It sorts designs by which alternative explanation each one can rule out. A case report rules out nothing; it generates a hypothesis. Allocation by lot is the only procedure that also rules out unknown confounders, because it pushes chance in between the groups.
| Design | Rules out | Stays open | Typical false conclusion |
|---|---|---|---|
| Case report, case series | nothing; the design describes a course of events | whether the course would have been the same without treatment | “It worked” |
| Laboratory experiment, cell model | interference from the rest of the organism | whether the mechanism matters in a living animal or human | reading a mechanism as proof of effect |
| Cross-sectional survey | nothing about the order in time | which came first, cause or consequence | reading an association as a cause |
| Cohort and case-control study | gross reversal in time, and some known confounders | unknown confounders, self-selection into the groups | “People who do X stay healthier, so X helps” |
| Randomised trial | known and unknown confounders at the start of the trial | whether the result holds for other animals, people and conditions | confusing efficacy under trial conditions with everyday benefit |
| Systematic review, meta-analysis | arbitrariness in the selection of studies | every weakness of the included studies | “It is a meta-analysis, so it is proven” |
The level is therefore not a verdict on the quality of the researchers. A carefully controlled cohort study can be more reliable than a badly conducted randomised trial, and a meta-analysis built from ten weak papers stays weak. The pyramid ranks designs, not individual publications.
How much bias a careless randomisation produces can be measured. An analysis of 2,443 randomised trials drawn from 228 meta-analyses in the Cochrane Library compared papers at low risk of bias with those at unclear or high risk. Trials without a traceable random allocation, without concealed group allocation or without blinding showed treatment effects that were on average some eight to thirteen per cent larger. The difference is not dramatic, but it is systematic and almost always in the same direction: in favour of the treatment under test.
Five things, on the other hand, randomisation does not repair: a group that is too small, an outcome nobody cares about, an observation period that is too short, a heavily selected population, and selective reporting of the results. These five points are precisely why GRADE can downgrade a randomised trial far enough for it to land on the same certainty level as an observational study.
In veterinary medicine, reporting on these points has only partly caught up. A review of the 2009 and 2019 volumes of a journal of veterinary anaesthesia found that full description of the randomisation rose from 13.6 to 85.7 per cent, and the reporting of a sample size calculation from zero to 20 per cent. Blinding stayed at around half of the papers and did not move in ten years. Not a single item reached the mark the authors had set as their target.
The pyramid suggests that observational data are fundamentally less reliable. A 2014 Cochrane methodology review examines exactly that: fourteen methodological reviews holding data from 1,583 meta-analyses across 228 conditions compared the effect estimates of randomised and observational studies addressing the same question. On average they did not differ meaningfully, and eleven of the fourteen reviews found no difference at all. The authors did not conclude from this that design is irrelevant, but that factors other than the label explain the discrepancies.
For some questions the observational study is in any case the only possible design: rare harms, long time spans, situations in which allocation by lot would not be defensible. A 2025 review therefore proposes replacing the pyramid with an amphitheatre for questions of post-authorisation medicine safety. All data sources stand in a circle around the question, and none automatically stands on top.
No. The finding holds for the average across many questions, not for the single study in front of you. With an observational study, what remains decisive is whether the groups being compared already differed before treatment, and whether that difference was balanced out. If nothing at all is reported on this, the result cannot be interpreted.
The softer the outcome, the more the expectation of the person doing the measuring comes into play. A pooled analysis of ten animal experiments with 2,450 animals in total compared blinded and unblinded assessment of the same experiments. Unblinded assessment exaggerated the treatment effect by 59 per cent on average. In most of these experiments the outcomes were subjective, that is, observations rather than measured values. That average, however, is carried by three experiments with a very large deviation; without them, the remaining seven leave an exaggeration of about one fifth, which is no longer statistically significant.
In small animal medicine this mechanism has been measured directly. In the placebo group of a registration study of 58 dogs lame from osteoarthritis, owners saw an improvement in 39.7 per cent of cases, and the examining veterinary surgeons, scoring lameness, in 44.8 per cent. Over the same period the force plate showed no change in 46 of the 58 dogs. In cats with joint pain, five studies covering 96 animals found the same pattern: 54 to 74 per cent of the placebo animals counted as a success in the owners’ judgement, but only 10 to 63 per cent by objectively measured activity.
Because it is not the animal that supplies the outcome, but the human being who judges it. Add to that regression to the mean: treatment usually starts when the complaints are at their worst, and a better phase often follows anyway. The two together produce a visible improvement without any active substance at all.
GRADE starts randomised trials at high certainty and observational studies at low certainty, then moves them. Five reasons lead downwards:
Three reasons, conversely, allow observational data to be upgraded: a very large effect, a clean dose-response relationship, and confounders that would have shrunk rather than inflated the effect found. An observational study can thus rise to moderate certainty and a randomised one fall to very low. Within GRADE, then, the pyramid is not a fixed shelf but a starting position.
Certainty of the evidence and strength of a recommendation are two different axes, and they often come apart. A systematic survey of 133 guidelines carrying 433 recommendations on acupuncture found 303 recommendations with a GRADE rating. Of those, 131 were worded strongly, and 104 of these strong recommendations rested on low or very low certainty. Emphatic wording is therefore no evidence of emphatic data.
In the classic drawing, the systematic review sits at the top. A much-cited proposal from 2016 takes it out of the pyramid and turns it into the lens through which the levels below are viewed. The reason is simple: a review generates no new data. It can only order and weight what is already there.
A stocktaking from the same year shows how necessary this correction is. Between 1991 and 2014 the number of systematic reviews published each year grew by 2,728 per cent, while all indexed papers together rose by 153 per cent. On antidepressants for depression alone, 185 meta-analyses appeared between 2007 and 2014. The author concluded that the great majority of the reviews being produced are redundant, misleading or shaped by vested interests.
In equine medicine the opposite problem is the rule: too little rather than too much. A 2024 systematic review of drugs against equine herpesvirus 1 screened 7,009 publications and kept nine; the risk of bias was moderate to high and the groups were small. A review of electrotherapies in horses, dogs and cats screened 5,385 publications, included 41 and arrived at an average of 3.7 papers per procedure, 20 of which carried a high risk of bias. The conclusions are correspondingly cautious: the electrotherapy team considers the evidence sufficient for not a single application in horses, dogs or cats, and the herpesvirus team saw at best a minimal or limited benefit for all the drugs examined.
The second fault line is the change of species. A systematic review compared six treatments for which a clear result existed in humans with the corresponding animal experiments. For corticosteroids after traumatic brain injury, the animal models showed a benefit and the clinical trials none. The drug tirilazad shrank the infarct area by 29 per cent in the animal model and improved neurological scores by 48 per cent, while stroke patients on the same drug fared worse than those without it.
Part of this discordance goes back to methods, not to biology. An overview of 31 systematic reviews of animal studies found that 29 per cent of the individual studies reported any randomisation at all, 15 per cent a concealed allocation and 35 per cent a blinded outcome assessment. A Swiss analysis of 1,277 animal experiment applications found the seven standard measures against bias described at all in only 2.4 to 19 per cent of the applications.
The following seven questions can be put to almost any report, usually to the abstract alone. They are no substitute for a formal appraisal, but they reliably separate the reports that show something from those that merely assert something.
Original analysis
| Review (field, year) | Starting pool and included studies | Certainty according to the authors | What does not follow from it |
|---|---|---|---|
| Drugs against equine herpesvirus 1 (horse, 2024) | 7,009 publications screened, 9 included | risk of bias moderate to high, small groups | that the drugs examined are ineffective: too little has been tested to decide that |
| Electrotherapies in horses, dogs and cats (2022) | 5,385 publications screened, 41 included, on average 3.7 per procedure | 20 of the 41 papers at high risk of bias | that a single positive paper establishes a procedure |
| Nutritional supplementation in unexplained infertility (human, 2019) | 5,663 records screened, 61 studies, 59 of them randomised | very low for every outcome | that many randomised trials automatically add up to high certainty |
| Animal-assisted therapy in dementia (human, 2019) | 9 randomised trials, 305 participants, one study with horses | very low to moderate, depending on the outcome | that an effect on mood carries over to quality of life |
| Parathyroid surgery (human, 2023) | 8 randomised trials, 447 adults | moderate for cure, very low for bone density | that one piece of work has a single certainty level valid for everything |
No, but it answers a different question. Allocation by lot is the only procedure that also balances unknown differences between the groups, which makes it the strongest protection against false conclusions about causes. It does not protect against groups that are too small, short observation periods or ill-suited outcomes. In an analysis of 1,583 meta-analyses, randomised and observational results on the same question did not differ meaningfully on average. For rare harms and long time spans, the observational study remains the only workable design.
It means that the true effect very probably lies clearly elsewhere than the reported result, in an unknown direction. The statement is therefore not a basis for a decision but a description of the state of research. One point matters here: very low certainty does not mean that the treatment does not work. It means that we do not know. In a 2019 review of nutritional supplementation for unexplained infertility, certainty sat at this level for every outcome, even though 59 randomised trials were included.
Because a single study cannot check its own result. Chance, local peculiarities of the population and choices made during the analysis all play a part in every individual paper, and only replication by independent groups separates signal from noise. A well conducted single study can still be the best that exists on a question. It then remains the best available basis, but with low certainty, not with high.
No, and no more so the other way round. A systematic review compared six treatments with a clear result in humans against the corresponding animal experiments and found several plain contradictions: one drug shrank the infarct area by 29 per cent in the animal model, while stroke patients on the same drug fared worse. An animal result justifies a hypothesis for the other species, nothing more. The testing has to take place in the target species and the target population.
By what is missing. If the text names neither the species studied, nor the number of participants, nor a comparison group, nor the observation period, then there is no basis for placing it at all. A second warning sign is the confusion of association and cause: wording such as “lowers the risk” presupposes a design that can test causes. A third is the surrogate outcome: an improved laboratory value is not the same as improved wellbeing.
Because nothing has replaced them on their particular question. The Cochrane methodology review comparing randomised and observational results still stands in its 2014 version; the compilation on agreement between animal experiment and clinical trial dates from 2007. In methods research a paper ages more slowly than in clinical research, because it examines procedures rather than substances. Where more recent figures exist, they have been given preference here.
Evidence Lab
Only about 5 per cent of therapies tested in animals ever obtain approval. What animal studies say about humans and horses, and what they do not.
MUMultiple speciesBmoderateT1hypothesis only Open access
Comparative Health
Radiocarbon data suggest the Achilles tendon core forms in growth and is barely renewed thereafter: what that means for healing, scar and the equine t
MUMultiple speciesBmoderateT2limited target data Open access
Comparative Health
Serum amyloid A rose about 227-fold in an equine experiment, haptoglobin only 1.1-fold. How inflammation starts, how it ends, what CRP and SAA measure
MUMultiple speciesBmoderateT2limited target data Open access
Once a month: new syntheses, checked studies, and what has changed in the state of the evidence.