"A new study says...." What you should know about those studies!

Editor's Note: When you see these three dots surrounded by a gray rectangle — 1 — you can click on it to get further information about the topic. Click a second time, and the message goes away.

This article was edited and updated on November 10, 2019.

What is the Scientific Method?

The scientific method is "a procedure that has characterized natural science since the 17th century, consisting in systematic observation, measurement, and experiment, and the formulation, testing, and modification of hypotheses." 2

To use the scientific method effectively, one needs to begin as a skeptic. All good science begins (and it must begin) with skepticism.

The definition of skepticism is "an attitude of doubt or a disposition to incredulity either in general or toward a particular object; the doctrine that true knowledge or knowledge in a particular area is uncertain; or the method of suspended judgment, systematic doubt, or criticism characteristic of skeptics."

In particular, "Scientific skepticism involves the application of skeptical philosophy, critical-thinking skills, and knowledge of science and its methods to empirical claims, while remaining agnostic or neutral to non-empirical claims (except those that directly impact the practice of science)."

Nothing is "true" until it is proven through these means. There is always a possibility for change, a new idea, a new concept, a new hypothesis — all of which means that one goes through the process again — and perhaps many times more.

What, exactly, is that scientific process?

It is a problem-solving approach, consisting of several steps:

The scientific method is rigorous and systematic, thus hopefully eliminating bias and other subjective influences. Through the process, it attempts to search, identify and measure or validate facts or cause-and-effect relationships. If all steps are adhered to, and sincere efforts are made to adapt and adjust when necessary, the final conclusions can be relied upon to be valid — until or unless someone else finds a different result. Then more research is required.

So how do I know if the scientific method has been used in a study?

You don't know unless you ask some very important questions.

Here are some of the questions you ask if you're reading a study, and/or you hope the journalist has read the study before using it on the news:

A Word about P-Hacking

P-Hacking is the "use of data to uncover patterns that can be presented as statistically significant without first devising a specific hypothesis as to the underlying cause." It's a little like putting the cart before the horse: Force the study to reveal an hypothesis which it was never intended to discover.

This can happen when the researchers are:

In other words, the data is there in basic numbers and statistics, but the interpretation of that data can be skewed because of bias or expected outcomes.

This definition of p-hacking comes from the online Urban Dictionary:

"[P-Hacking is the] manipulation of statistics such that the desired outcome assumes 'statistical significance', usually for the benefit of the study's sponsors. Usually done by ex post facto choice of significance labels and simple reporting of results as being conclusive regardless of calculated p-value. This depends on the public's general lack of understanding of statistical measures and press non-reportage of details."

Note the final comment about "press non-reportage of details." When you hear about a study on the news, it's usually a very short article, lasting only a minute or two. You are given very few details of how the study was conducted, and thus you do not know how accurate the "conclusions" are.

Do you know what a predatory journal is?

Sounds terrible, doesn't it? Predatory journal. Well, it's exactly what it sounds like: not good.

"Predatory Journals take advantage of authors by asking them to publish for a fee without providing peer-review or editing services. Because predatory publishers do not follow the proper academic standards for publishing, they usually offer a quick turnaround on publishing a manuscript." (See the Wikipedia article about predatory Journals: https://en.m.wikipedia.org/wiki/Predatory_publishing).

Whenever possible, of course, you must choose to read the top professional journals, or check to see in what journal the study was published. My son, who used to be a registered massage therapist and is now science writer, tells me this: He created a top ten list of the best journals for pain and injury science to trust and to use. He put these journals on his list: Archives of Physical Medicine and Rehabilitation, Pain, Physical Therapy, Stroke, Clinical Rehabilitation, Spine, Lancet, British Medical Journal, Journal of the American Medical Association, and Journal of Physiotherapy.

If you're not a scientist, how can you possibly know if the study is of value?

The more technical the research, the harder it is to understand. For those of us who are not scientists, we have to rely on the source of the information as well as read the details of the study carefully.

Unfortunately, the news media may be your worse source — or the company who's trying to sell you a product. As with everything, you have to be a savvy consumer.

There are some websites where you can find medical studies discussed in laymen's terms and the source is reliable. One is simply named Cochrane which is mostly about medicine and health. From their homepage you can go to the Cochrane Library and read about older studies.

If you want to see a humourous but serious look at this topic, watch this video with John Oliver.

Studies have to have good science behind them.

If you see a headline like this — "Study Shows that Eating Chocolate Every Day will Help you Lose Weight" — it's probably worth checking that carefully. A great many of the "study reveals" stories on the evening news look for a good headline, but there's not always good science behind them. Before allowing yourself to believe it — and perhaps even buy the product or change your eating or fitness habits — be sure to check its authenticity.

I am a BCRPA-certified fitness instructor in Vancouver, BC. I teach four classes at the West End Community Centre in Vancouver, BC, mostly designed for the older adult. The Inevitable Disclaimer: Everything published here expresses only my opinion, based on my training and research. What you do with the information is entirely your own responsibility. I am not liable for any injury you suffer that seems to be related to anything you read here. Always consult your doctor before beginning an exercise program. For other articles, return to the table of contents.

★ ★ ★

  1. These three dots behave exactly like a footnote. Click on them and you will get more information about the topic. 

  2. From Wikipedia.