How to Spot Bad Science

Question: I like to do my own research on difference products or treatments for my arthritis but I don’t have a science background- how can I tell if what I’m reading is from a trustworthy source or if the study I’m reading is of good quality?

Unless you’ve studied it, most of us are never taught how to evaluate science (e.g., quality of scientific studies or scientific claims) or how to determine the good from the bad. Yet it is something that dictates every area of our lives- we are exposed to claims, “facts”, and statistics every single day online, TV and on social media. Being able to evaluate the quality of evidence behind a scientific claim is important. Being able to recognise bad science reporting and interpretation, or faults in scientific studies is equally important.
This article cannot possibly cover all aspects of spotting bad science- there is just too much to cover! However, the lessons and hot tips below will certainly cover enough to improve your ability to spot bad science.

Lesson number 1 – Bias and preconceived beliefs

When comes to researching information about a certain product, medication, therapy or treatment, be aware of both your personal and the author/s biases. The two biggest ones are:
Confirmation bias and belief perseverance.
Confirmation bias a bias in which people seek out and recall information that supports their preconceived beliefs
Belief perseverance is the rejection of information that could disprove beliefs

Preconceived beliefs are often implicitly and unconsciously developed, and may be informed by culture/religion, schooling/education, generational, family, friends, and society. Regardless of where these beliefs came from, it is important to be open to alternative scientific outcomes or options, and allow your beliefs to be challenged and even changed or “up graded” if they are incorrect or “out of date”.

Furthermore, when reading scientific studies and particularly blogs, articles, and information on other social media and websites, it is important to consider if the author/s is practicing confirmation bias or belief perseverance- see hot tip below.

Hot Tip: is this author or study paid or endorsed? Do they have any sort of relationship/conflict of interest with a product/treatment/therapy where they could benefit financially or politically? Do they have polarized or ridged views? Does the paper talk objectively about the results or outcomes, mention both significant and non-significant findings, and/or the study limitations? If not, the paper and/or authors may not be telling the whole truth and are shaping the results to fit in with their preconceived beliefs.

Lesson number 2 – ways to spot bad science

The “A Rough Guide to Spotting Bad Science” infographic outlines and explains 12 ways to spot bad or fake science. You can apply this to scientific articles to claims and ads you see on the internet and TV. However, please note that this is not a comprehensive overview, nor is it implied that the presence of one of the points noted automatically means that the research should be disregarded. This is merely intended to provide a rough guide to things to be alert to when either reading science articles or evaluating research.

Lesson number 3 – types of scientific evidence

The “A Rough Guide to Types of Scientific Evidence” infographic outlines and explains different types and strength of scientific evidence. As you can see, the best articles to read are systematic reviews followed by randomized control trials. Animal and cell studies (in vitro studies), and anecdotal and expert opinions are the least “trustworthy” or weakest form of evidence. Having said that, animal and cell studies are not bad, in fact they can be very useful in testing the safety, toxicity and efficacy of a drug, and opinions/anecdotes aren’t necessarily bad either, however, as mentioned earlier, they can be biased or there can be a conflict of interest. Each types serves a purpose, however, it is best practice to formulate opinion on the outcomes of high quality studies that use human participants.

Hot tip: when researching, try typing into the search engine “systematic review on….” Or “randomized control trial on …..” or some variation including those key words, and for more recent findings, ensure to change the date range to exclude studies that are more than 10 years old. See below using Google Scholar.

Lesson number 4 – common place pitfalls in reasoning

Fallacies are common errors in reasoning that will undermine the logic of your argument, opinion or choice. Fallacies can be either illegitimate arguments or irrelevant points, and are often identified because they lack evidence that supports their claim. See below for some common examples.

Emotional reasoning fallacy
The emotions of an individual have a significant effect on reasoning performance. The Emotional reasoning fallacy is the error of using our emotions as guides for evaluating the validity of a claim. For example: “I feel afraid, so I must be in a dangerous situation.” It’s like we’re saying to ourselves “I feel, therefore it is” – rather than looking at what real evidence there may be.

Bandwagon fallacy
is the error of assuming a claim is correct just because many people believe it. For example: “over 1000 people say their pain vanished when they started taking X…”

Not me fallacy
Error of believing we are immune from thinking errors

Bias blind spot fallacy
Lack of awareness of our biases, couple with an awareness of others’ biases. We spot other peoples biases but either ignore or can’t see our own.

Hot tip: it’s important to practice a little introspection and objectivity when committing to a belief, reason for trying a product/therapy/treatment, or forming an opinion. Try not to go with the crowd or popular thinking “just because”- instead, investigate for yourself. Also, try and not let your emotions get the best of you and guide your decision making, and always be open to being wrong and/or changing your views.

I hope this Ask Kat has been helpful. If you have any questions on the above information, please email them to kkeane@arthritisnsw.org.au.

Author: Kat Keane, ANSW Health Education
Date: 14/9/2021

References

Infographics: Compoundchem
https://i1.wp.com/www.compoundchem.com/wp-content/uploads/2014/04/A-Rough-Guide-to-Spotting-Bad-Science-2015.png?ssl=1
https://i2.wp.com/www.compoundchem.com/wp-content/uploads/2015/04/A-Rough-Guide-to-Types-of-Scientific-Evidence.png?ssl=1

 

If you have a question about arthritis and related topics that you want answered, please email Kat for her to answer in the next eNews. Kat is ready and waiting.