Scientific Research: The compilers, commentors and the researchers

In this article, I talk about the biases that can arise from so called scientific research.

A disclaimer, this article stands at about 3700 words, and is by no means an easy read in terms of the concepts thrown out and the length itself. Needless to say, this is not for everyone. But if you’re a sceptical empiricist like myself, hopefully you will find it useful.

We often hear claims about health that spark interest and debate. For example, author Gary Taubes writes about how the calories we eat are important to overall health and is a prime contributor to obesity. He argues for a low carbohydrate high meat diet. While I am not here to bash his claims (many others already have), I am wary of such simplistic claims, based upon such flimsy research (look up his bibliography in the book ‘Good Calories, Bad Calories’). Very often, the solution is not very simple; the human body is far from a simple machine.

Ultimately, the conclusion of this article is that, you have to SUSPEND JUDGMENT regarding many claims of so called ‘scientists’. We must look out for sound methodology in performing experiments, and ultimately the true results that are delivered in the real world.

If you cannot satisfy these two criteria: Ignore the claim entirely. Remember, you are allowed to say, “I DON’T KNOW.”

The four main areas which I touch on in this long article will be:

1. The vast amount of data available
2. The complexity of the human body
3. The nature of reporting: the narrative fallacy and oversimplification
4. The veracity of the research itself

Let’s get started.

The Mountain of Data/Anecdotes/Evidence and just plain junk and nonsense

Let’s face the fact that we have a lot of information in today’s world. With the rise of the internet and the lowered costs of publishing, content producers can basically be anyone. In the case of scientific research this is problematic of course.

There are 3 main problems presented by this huge pile of data. I will assume here that the interest of the reader is to get the relevant facts in order to be able to make an informed decision about how to act to improve one’s health/well-being (positive aspect).

First, the information that we may need to know may not be found. This is where search comes in, but then again, there is a limit to how much you can search for. Very often, things do not pop up when we need it the most. For example, we receive novel evidence of how certain plastics used in manufcature can be harmful to us. As pendantic practitioners of the scientific method (aren’t we all?), we go about searching for evidence to disprove this theory.

Remember the Lessons of Karl Popper, most importantly the need for a Theory to be possibly falsified. In this case, we need to see if there is conflicting evidence to our readings so that we know the conditions by which they are valid and therefore, when they are invalid.

In doing so, however, we find that there is this huge mountain of data. Millions of volumes waiting to be combed with potential solutions and proofs of our misconception. This is a big problem.

Then comes the 2nd problem. Frankly, in this huge pile of data, half of it is irrelevant, 40% is crap, 5% agrees with you, 4.9% presents long waffly arguments which we can safely disprove, and the remaining 0.1% is truly something worth looking at. Finding those piece is certainly a big problem, but having the will and time to browse through all the rubbish to get to them is even harder.

And thus leads to our 3rd problem, is that with so much information, it is basically impossible to get a ‘thorough’ plan.

Say you want to spend some money on anti-oxidant supplements. Anyone who buys such supplements would realise a huge discrepancy between various brands. You thus set out to minimise your costs on such supplements. One company claims one thing, another claims that their pills have a magical ingredient in them, and you’re just plain confused. We as humans obviously did not evolve to handle such mountains of information. We thin-slice, we Blink.

And yet, we face a Paradox of Choice, that with so many brands to choose from, how would we possibly know that we are getting the best deal? Furthermore, what if one company suddenly has a sale which cuts your costs in half, while you have already decided to buy from another company? And why not source overseas, perhaps even with the shipping factored in, your purchase would be even less? Finally, does it even matter that you get the more expensive one? Is it necessarily better?

These are not rhetorical questions, and different people can answer them differently. The most straightforward case would be the rich man, who would simply ignore the cost. But even he would have to consider the efficacy of the supplements. In this case, with so many choices and sources, it becomes easy to imagine a perfect brand of supplement, and the fact that you did not get it is a failure on your part.

The Complex Human Body

To further exacerbate the problem of the huge piles of data, the data touches upon many different variables.

The human body is a complex machine, and very often (almost always), effects are not caused by a single substance and usually take years for symptoms to show. Even something like Mercury poisoning does not happen overnight like some poisons like arsenic, it occurs from a slow accumulation of environmental methy-mercury from the environment, and depending on the levels of exposure, can take tens of years to really show symptoms.

The one example I want to raise here is the issue of fat intake in the diet. This has been an area of fervent debate for a few decades. For a good review of dietary fats, Casey Butt’s article series is a good read. If you lived through the 80s to present, you would have heard of the times when all fat was bad, which evolved into saturated fat is bad, and now that trans fats are bad. This has led to the either confusion, or people throwing up their hands in digust of the scientific method (often people who don’t know anything about science in the first place).

There was nothing wrong with the scientific method, and in fact there were studies which showed correlates between saturated fat and heart disease. But there were also others which didn’t. On top of that, we have to insert the huge factor of genetic variability.

To give one example, insulin sensitivity seems to vary across ethnic groups. People who descended from Africa seem to have on average higher insulin sensitivity. But then again, the issue with fat does not revolve around a single hormone. In fact, there are hundreds of different hormones controlling everything from fat digestion, to fat storage, to fat mobilisation and oxidation. How these hundred hormones react to a given ration of saturated fatty acids is a hugely complicated matter.

Furthermore, you obviously don’t die from eating a single fatty meal, and not even after a thousand. It is a long cumulative effect. Therefore, any real credible study will have to be done over a long period of time.

Finally, we need to fact check with the real world. One of the countries with the highest rate of heart problems is India, and their diet probably conforms to a high saturated/trans fat intake. Why are the french not really affected by this though? And are the Inuits not getting these problems because of the low saturated fat intake or due to the high Omega 3 intake from fish. Are Indians more prone to disease from eating saturated fat as compared to the French, or is it a real correlate? We frankly do not know a lot about this, and shooting blind correlates do not work.

There lies a problem with human nature, that we like to attach casual relationships to events. We think that drinking coffee before a test improves results and yet have no way of proving it. While in practice, you should continue to do that, since even if it is a placebo effect, it is possitive, but trying to draw conclusions from that and then saying this is universally applicable to everyone is stupid at best and harmful at worse.

Take the saturated fat episode. The fear of taking saturated fat led to McDonald’s using partially hydrogenated vegetable oils instead of beef tallow to fry their french fries. This basically put in a whole load of trans fat into the mouths of those who ate french fries (Trans fat are bad in any case, no questions asked and there is enough research to support that). Or, it led people to eat high carb low fat diets, of which some people started eating more sugar, which led to a lot of problems.

Yet, trying to diagnose this problem is hard. Sugar per se does not do you much direct harm, it is the underlying conditions, and also, the nature of the experiment. I will return to this in the last part of this article.

Hence, the take-away from this part is: Science is Complicated. The Human Body is Complicated. Most people think they know more than they really do. When in doubt, which should be very often, say “I Don’t Know”.

By now you’re probably wondering, “So what do I do then?” Bear with me, for I shall give recommendations at the end of this (monstrously long) article. But first, I shall touch on why you should not listen to anything from the mainstream media.

The Problem with Reporting Scientific Facts: Multiple Fallacies

Here comes my greatest pet peeve, and I will state my recommendation right now extremely clearly: Do not listen to ANYTHING in the mainstream media unless they have proper citations that can be referenced. Even then, one should take considerations to the current body of knowledge and the real world results and compare these to what the study claims.

My main problem here is that it is human nature (at least instinctively) to simplify things for better understanding. We tend to remember things better when they are told as a narrative; a story which we can tell ourselves and one that we believe in. That means that we scorn the abstract; anything that cannot be put into a coherent (one that fits our beliefs) narrative is rejected.

This is made worse, due to a second problem with human nature, unfortunately due to our evolution once more. It is the fact that we over-estimate the probability of events with negative outcomes, but only those which directly affect ourselves. For example, just because there are cars one the road and accidents have occurred, it does not therefore mean that every time your children are near a road, they will be run down by a car. Or worse still, some parents don’t let their children out after 5pm due to the chance of them being abducted by some random stranger.

Well, the chances of that are 1 in 100000, but the human brain wasn’t designed to relate to probability; after all, we cannot ever visualise probability because at such low values, it is largely dependent on what does not happen. Instead, we turn to our gut instincts and judge the situation based on the worst-case scenario. It must have been a smart evolutionary move for our brains to evolve to rather mistake a shadow for a tiger than a tiger for a shadow; we would be dead if the latter were the case. We then take this notion in basically everything we do, since our DNA (which took 100 million years to evolve) hasn’t changed in the last 20 years (when the boom in information occurred).

The mainstream media knows this, not that they understand it this way, but they understand that the way to get people’s attention is to create something sensational. After all, who wants to know that losing weight is a matter of reducing calories in a sustainable fashion and stick to a resistance training program over a long duration (which is the truth) when they can lose 20lbs in 7 days using some miraculous product. The problem which I have is when cases like the above come out, giving people misconceptions of what to expect, and then becoming disappointed in their failings. The problem is that sensationalism is usually far from the truth.

Of course, claims aren’t always that radical (some are), but there are some which have the potential to mislead people. There are two examples I would like to raise. First, is the ‘easily’ (they are all easy to debunk in the end) debunked one. I recently received a link to a supposed cure for cancer, the sour sop. The article was a long waffly piece denouncing chemotherapy and telling people to trust in the sour sop to cure cancer, asking why hadn’t anyone thought of it before. Well, chemotherapy actually increases the chances of recovery or spontaneous remission, so its your best bet. Ultimately, these sorts of articles cite no scientific evidence or flimsy anecdotal evidence and any smart reader would ignore them outright.

Then there is the kinds of articles (and books too) which have such citations. The most recent example of a false article with plenty of citations is the recent article in TIME about ‘Why exercise won’t make you thin‘. In this article, John Cloud cites several studies done by various researchers. One of his arguments revolve around the concept that exercise increases the appetite of people and they end up overcompensating for the calories which they burned during exercise. Which is true in some cases, but it comes under the assumption that people are eating ad libitum and thus not controlling their caloric intake. Not to mention the fact that some people don’t overcompensate for these calories; those who either don’t feel hungry or don’t have that ‘I’ve exercised so I deserved that cupcake’ mentality. Finally, it virtually ignores the other benefits of exercise. I can go on about his, but plenty of people have already bashed him for that article, so I will end here.

The main point is that just because someone has the ‘scientific research’ to back him/her up, does not lend credibility to the work until those sources have been checked and confirmed for their validity. This will be the last problem which I tackle.

The Method of the Researchers and the veracity of their research

Alan Aragon has written an article titled “Commercial Bias in Scientific Research” that is definitely worth your time.

In it, he talks about various biases, and the ones that I do want to note are the publication and funding bias. Essentially, what he says is that only a minority of scientific papers are actually published, and of those that do, there are some which are produced with the funding of various companies which may have a vested interest in the product. I will not comment further, and I urge you to take a look at that article some time.

What he says is true though, and this probably plays into another fallacy: the fact that even today, many scientists formulate a conjecture and then try to prove it right, when in fact they should try to prove it wrong. Even the best theories may be falsified, just as Einstein’s special relativity does not apply to us at our measly speeds. This is worse so when it comes to the human body, when the complications as described above need to be taken into account. (When there isn’t say a universal law of gravity to turn to as a benchmark) To put it simply, the Method Matters.

Let us generate a hypothetical study with the hypothetical title, “Administration of high omega-3 dosage reduces LDL cholesterol by 20%”. Sounds alright, but we need to check the methods. In this case, let’s say that the researchers included the statement, “50 sedentary caucasian adults were recruited for the study, of which they were split into 3 groups, a control group, a group given 6g daily does of generic fish oil, and a group given 20g daily does of generic fish oil. Measurements were taken on a weekly basis for 3 months.”

They then perform the experiment and discover a reduced cholesterol in the group taking 20g fish oil a day by 20%+-8%, while the group that has 6g a day has a 10%+-6% reduction, with the control having a negligible difference. The researchers then conclude that a 20g does of fish oil is potentially useful for the reduction of LDL cholesterol and more research needs to be done.

That is perfectly alright. But I am not going to take this very seriously for several reasons. First, the sample size is small; it may not reflect that of the general population. Second, there were only 3 separate groups; how do we know whether a 12g dose might yield the same effect as the 20g dose? And third, other factors were not controlled. In this case (and in many), food intake was not controlled, neither was activity. Weight and body fat levels were not tracked. Given that we do have a positive correlation of lower body fat levels and lower overall bloody cholesterol levels, it is important that the subjects were at the same ‘fatness’, so that any possible benefit was due to the fish oil and not a reduction is body weight. Given that the adults were sedentary to begin with, this could have been the case, and because of the low sample size, 1 person who ‘happened’ to adopt a healthier lifestyle during the 3 months of the study could drastically affect the sample.

This brings me to highlight the error bars, whereby we see a 20%=-8% reduction. That means that the lowest reduction in LDL cholesterol could only have been 12% while the highest is 28%. Why is this the case? The researchers do not know. All they can do is look at data and comment.

Whereas, if you look at the Aspartame studies I cited in my post “A case for Zero Calorie Sweeteners“, they do not fall for most of these pitfalls. Or to put it slightly differently, you do not see people in the real world dying from drinking a litre of diet soda a day, and thus the likelihood of you dying from drinking a can of diet soda a day is pretty slim.

If up to now you are dizzy with all the pedantic talk about science and its methodology, you obviously know why most people cannot do science in its most pristine form. It is inherently a subject (at least at higher levels) dominated by extreme randomness and experimentation. We literally conduct experiments, test our hypothesises, tally count of the number of times it was proved right and proved wrong, and then bet on the likely scenario.

Most people cannot do that, and that includes journalists. They are particularly subject to the narrative and causality fallacies precisely because they job entails the necessary condition of informing through narration Not to mention the fact that most in the mainstream media (not science journals) do not understand science. The hypothetical study may be interpreted by a TIME journalist as a 20g dose of fish oil reduces heart attack rates, since lowered LDL cholesterol is correlated (albeit not absolutely) with reduced incidence of heart attack amongst the population. He/she would then go on to recommend that fish oil should be taken in copious amounts to ‘boost heart health’. We have strayed far from the original researchers conclusion of “more research is required” (a euphemistic but honest way of saying ‘I don’t know’).

Finally, I would like to talk about the researchers themselves. A researcher is obviously a human being, and his or her research is based upon a conjecture which is formulated bearing in mind certain boundary conditions. These conditions are tested and variables are subject to scrutiny in hopes of coming closer to the truth. This is the (close to) ideal scenario.

However, like I’ve stated earlier, the truth sometimes needs more resources than available. For the aspartame studies, many were conducted by state departments because of the long term (20 odd years) nature and high sample size (into the thousands) of the study. Yet, it is only in matters of such controversy whereby state departments are employed to the task. Who would want to go through the legislative paperwork to spend the state’s money on research on the new drug by some obscure supplement company.

If you’d like to be cynical, it is very possible that such a company, if big enough (like many are) could sponsor a team of researchers to test it for them. Therein lies the possible bias, but even so, a single positive does not say much; we need to count the misses as well.

Then comes to our final group. If you’re wondering where all the research papers are coming from (after all, not everybody is doing research), you just need to turn to the universities of the world. Many studies, arguably the majority, are done by graduate and sometimes undergraduate students looking to prove a hypothesis for their own thesis paper. We obviously see the problem, since college students would try to delve into ‘novel ground’ (which hasn’t been adequately tested), and lack the funding to do large scale or long term projects (semester only lasts so long), their research should be taken with a grain of salt.

Again, the cynic would say, a new magical ingredient is found, another student gets an A, your pockets become slightly shallower.

What to Do

In the light of all that I’ve said above, I am slightly embarrassed to say that the recommendations are the shortest of the all the sections, mainly because they are so straightforward. This does not mean however that they are easy to practice.

Here they are:

1. If you are a consistent practitioner of empirical scepticism, continue what you are doing
2. If you are not the above, when you hear anything in the mainstream media, don’t listen to it at all
3. When in doubt, make it known; ask for Permission to Withhold Judgment with the phrase, “I’m sorry, I Don’t Know.”
4. Review all data as far as possible. Read broadly and hope that you discover somewhere in which you are wrong.
5. Always review the theoretical data with accordance to the real world experiences and not do it the other way around and try to fit the real world into your data
6. Practice what has been proven to be consistently correct in the real world (losing fat and exercising to maintain heart health for one)

Thus I end this monster of a blog post, and thank you for reading it.

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

Tagged as: General, health, media, nutrition, research