Why RCTs Are Not Always Conclusive

Randomised Controlled Trials (RCTs) are viewed as the gold standard to scientific investigation - and in many ways they are. However, RCTs are sometimes less conclusive than you might think, especially when considered in isolation and interpreted in overly reductionist ways. Why? Unfortunately, because biological systems are often complex machines, and looking at one variable in isolation can be misleading unless you consider it in the context of the system as a whole.

Unfortunately, this means science is a long, hard process requring huge amounts of RCTs (and observational studies) to slowly work out how the whole system is working, and how individual variables, like a particular vitamin or mineral an influence that system. Many bloggers about testosterone, and even some researchers, seem blind to this complexity.

To give a simple example. Let's consider an RCT that looked to explore the impact of Substance A on testosterone. Let's assume they have a control group, do proper blinding, have a good sample size etc. So they run the trial and find out that Pill A has no impact on testosterone. It seems conclusive - Substance A is unrelated to testosterone. They have tried it, it failed, case closed.

However, consider if the production testosterone required both the Substance A and an unstudied Substance B to be produced. Of course, in isolation, Substance A isn't going to show an impact, because the other requirement was missing. Substance A may be neccessary but not sufficient to boost testosterone. The testosterone factory required two inputs rather than one.

Unfortunately, in reality our testosterone factory requires much more than two inputs. Potassium, zinc, copper, magnesium and a whole host of other factors play a role.

Another consideration is whether Substance A has a simple linear relationship with testosterone (i.e. the more substance A the more testosterone), whether only sufficiency is required (i.e. if there is enough - it works like a light switch), or whether there is an inverse U relationship (you can have too little or too much, but there is a sweet spot in the middle).

If the system works according to a sufficiency model - this may mean if you are deficient in Substance A then testosterone production shuts down, but if you already consume over a certain level testosterone production may take place. Perhaps in our RCT the population being studied was already above sufficiency, so provision of additional Substance A did not help.

If there is inverse U relationship, there is a relationship between Substance A and testosterone, but you have to hit a sweet spot to see it. Perhaps in the study, the group were already past the peak point, or even the dosage was so high that participants went from one side of the U to the other.

It's important to bare all this in mind when interpreting RCTs and different results.