"A new study says...." What you should know about those studies!

Editor's Note: When you see these three dots surrounded by a gray rectangle — 1 — you can click on it to get further information about the topic. Click a second time, and the message goes away.

What is the Scientific Method?

The scientific method is "a procedure that has characterized natural science since the 17th century, consisting in systematic observation, measurement, and experiment, and the formulation, testing, and modification of hypotheses." 2

To use the scientific method effectively, one needs to begin as a skeptic. Nothing is "true" until it is proven through these means. There is always a possibility for change, a new idea, a new concept, a new hypothesis — all of which means that one goes through the process again — and perhaps many times more.

What, exactly, is that process?

It is a problem-solving approach, consisting of several steps:

The scientific method is rigorous and systematic, thus hopefully eliminating bias and other subjective influences. Through the process, it attempts to search, identify and measure or validate facts or cause-and-effect relationships. If all steps are adhered to, and sincere efforts are made to adapt and adjust when necessary, the final conclusions can be relied upon to be valid — until or unless someone else finds a different result. Then more research is required.

So how do I know if the scientific method has been used in a study?

You don't know unless you ask some very important questions. Here are some of the questions you ask if you're reading a study, and/or you hope the journalist has read the study before using it on the news:

  1. Who sponsored the study?
  2. Who or what was studied (people, mice, etc.)?
  3. How many were studied? In other words, were there large enough numbers to make the statistical results worthwhile?
  4. How long did it last?
  5. Was there a control group (a group that did not do the things that the other group did)?
  6. How were the results measured?
  7. What were the conclusions of the researchers, and how did they arrive at their conclusions?
  8. What kind of a research was it (anecdotal, double-blind placebo, experimental, clinical trial, or a meta-analysis)?

A Word about P-Hacking

P-Hacking is the "use of data to uncover patterns that can be presented as statistically significant without first devising a specific hypothesis as to the underlying cause." It's a little like putting the cart before the horse: Force the study to reveal an hypothesis which it was never intended to discover.

This can happen when the researchers are:

In other words, the data is there in basic numbers and statistics, but the interpretation of that data can be skewed because of bias or expected outcomes.

This definition of p-hacking comes from the online Urban Dictionary:

"[P-Hacking is the] manipulation of statistics such that the desired outcome assumes 'statistical significance', usually for the benefit of the study's sponsors. Usually done by ex post facto choice of significance labels and simple reporting of results as being conclusive regardless of calculated p-value. This depends on the public's general lack of understanding of statistical measures and press non-reportage of details."

Note the final comment about "press non-reportage of details." When you hear about a study on the news, it's usually a very short article, lasting only a minute or two. You are given very few details of how the study was conducted, and thus you do not know how accurate the "conclusions" are.

If you're not a scientist, how can you possibly know if the study is of value?

The more technical the research, the harder it is to understand. For those of us who are not scientists, we have to rely on the source of the information as well as read the details of the study carefully.

Unfortunately, the news media may be your worse source — or the company who's trying to sell you a product. As with everything, you have to be a savvy consumer.

There are some websites where you can find medical studies discussed in laymen's terms and the source is reliable. One is simply named Cochrane which is mostly about medicine and health. From their homepage you can go to the Cochrane Library and read about older studies.

If you want to see a humourous but serious look at this topic, watch this video with John Oliver.

Studies have to have good science behind them.

If you see a headline like this — "Study Shows that Eating Chocolate Every Day will Help you Lose Weight" — it's probably worth checking that carefully. A great many of the "study reveals" stories on the evening news look for a good headline, but there's not always good science behind them. Before allowing yourself to believe it — and perhaps even buy the product or change your eating or fitness habits — be sure to check its authenticity.

I am a BCRPA-certified fitness instructor in Vancouver, BC. I teach four classes at the West End Community Centre in Vancouver, BC, mostly designed for the older adult. The Inevitable Disclaimer: Everything published here expresses only my opinion, based on my training and research. What you do with the information is entirely your own responsibility. I am not liable for any injury you suffer that seems to be related to anything you read here. Always consult your doctor before beginning an exercise program. For other articles, return to the table of contents.

★ ★ ★

  1. These three dots behave exactly like a footnote. Click on them and you will get more information about the topic. 

  2. From Wikipedia.