Editor's Note: When you see these three dots surrounded by a gray rectangle — 1 — you can click on it to get further information about the topic. Click a second time, and the message goes away.
What is the Scientific Method?
The scientific method is "a procedure that has characterized natural science since the 17th century, consisting in systematic observation, measurement, and experiment, and the formulation, testing, and modification of hypotheses." 2
To use the scientific method effectively, one needs to begin as a skeptic. All good science begins (and it must begin) with skepticism.
The definition of skepticism is "an attitude of doubt or a disposition to incredulity either in general or toward a particular object; the doctrine that true knowledge or knowledge in a particular area is uncertain; or the method of suspended judgment, systematic doubt, or criticism characteristic of skeptics."
In particular, "Scientific skepticism involves the application of skeptical philosophy, critical-thinking skills, and knowledge of science and its methods to empirical claims, while remaining agnostic or neutral to non-empirical claims (except those that directly impact the practice of science)."
Nothing is "true" until it is proven through these means. There is always a possibility for change, a new idea, a new concept, a new hypothesis — all of which means that one goes through the process again — and perhaps many times more.
What, exactly, is that scientific process?
It is a problem-solving approach, consisting of several steps:
- identifying and defining a problem;
- accumulating the relevant data;
- formulating a tentative hypothesis;
- conducting experiments to test the hypothesis;
- interpreting the results objectively; and,
- repeating the steps until an acceptable solution is found.
The scientific method is rigorous and systematic, thus hopefully eliminating bias and other subjective influences. Through the process, it attempts to search, identify and measure or validate facts or cause-and-effect relationships. If all steps are adhered to, and sincere efforts are made to adapt and adjust when necessary, the final conclusions can be relied upon to be valid — until or unless someone else finds a different result. Then more research is required.
So how do I know if the scientific method has been used in a study?
You don't know unless you ask some very important questions.
Here are some of the questions you ask if you're reading a study, and/or you hope the journalist has read the study before using it on the news:
Who sponsored the study? Most of us know the history of the tobacco industry who knew for years that smoking was harmful to one's health but did not reveal that information to the public. Clearly, companies and corporations have agendas: they want to sell their product or promote their technique. After reading a study that says for ultimate health, you must eat 50 pounds of Kale a day, it might be worthwhile to see if a company who sells kale might have sponsored the study! (Note: Take a look at the section below titled "Predatory Journals.")
How many were studied? In other words, were there large enough numbers to make the statistical results worthwhile? The smaller the number of people studied, the less likely the results will hold up if larger numbers are used.
How long did it last? Studies of large groups of people over many years are more reliable than those that are done in a short-term analysis, particularly if one is attempting to determine how age factors into a condition.
Was there a control group? A control is a group that did not do the things that the other group did. One group takes the "real drug" and another group takes a placebo or does nothing differently than they were currently doing. Then they compare the two groups for similarities and differences.
How were the results measured? Sometimes results are very difficult to measure and sometimes these measurements are difficult for laymen to interpret. If you find that you do not understand how the results were measured, you may not be able to accurately interpret the data.
What were the conclusions of the researchers, and how did they arrive at their conclusions? Do not automatically trust conclusions and abstracts. You may need to read the entire study.
What kind of a research was it? There are different kinds of research: ANECDOTAL = personal accounts; DOUBLE-BLIND PLACEBO = two groups taking a "pill" where one is the pill that is being tested and the other is a placebo (and should have no effect); EXPERIMENTAL = testing subjects and their reactions (could be humans or laboratory animals); CLINICAL TRIAL = people with a particular health condition take a drug to see if it will help their condition; META-ANALYSIS = checking many studies to see if there are commonalities in the results. As you might guess, anecdotal is one of the least scientific methods of gathering data, even though it can be useful, and meta-analyses are the most likely to produce accurate data because it looks at many studies, large numbers of people, and over a longer length of time.
A Word about P-Hacking
P-Hacking is the "use of data to uncover patterns that can be presented as statistically significant without first devising a specific hypothesis as to the underlying cause." It's a little like putting the cart before the horse: Force the study to reveal an hypothesis which it was never intended to discover.
This can happen when the researchers are:
- not sure what their data represents;
- biased about the data; or,
- attempting to 'force' the data to fit their desired results (e.g. A company wants to prove that its product doesn't cause cancer. If you begin with an hypothesis and you can't make the data "fit" — then you manipulate it to make it fit your hypothesis.)
In other words, the data is there in basic numbers and statistics, but the interpretation of that data can be skewed because of bias or expected outcomes.
This definition of p-hacking comes from the online Urban Dictionary:
"[P-Hacking is the] manipulation of statistics such that the desired outcome assumes 'statistical significance', usually for the benefit of the study's sponsors. Usually done by ex post facto choice of significance labels and simple reporting of results as being conclusive regardless of calculated p-value. This depends on the public's general lack of understanding of statistical measures and press non-reportage of details."
Note the final comment about "press non-reportage of details." When you hear about a study on the news, it's usually a very short article, lasting only a minute or two. You are given very few details of how the study was conducted, and thus you do not know how accurate the "conclusions" are.
Do you know what a predatory journal is?
Sounds terrible, doesn't it? Predatory journal. Well, it's exactly what it sounds like: not good.
"Predatory Journals take advantage of authors by asking them to publish for a fee without providing peer-review or editing services. Because predatory publishers do not follow the proper academic standards for publishing, they usually offer a quick turnaround on publishing a manuscript." (See the Wikipedia article about predatory Journals: https://en.m.wikipedia.org/wiki/Predatory_publishing).
Whenever possible, of course, you must choose to read the top professional journals, or check to see in what journal the study was published. My son, who used to be a registered massage therapist and is now science writer, tells me this: He created a top ten list of the best journals for pain and injury science to trust and to use. He put these journals on his list: Archives of Physical Medicine and Rehabilitation, Pain, Physical Therapy, Stroke, Clinical Rehabilitation, Spine, Lancet, British Medical Journal, Journal of the American Medical Association, and Journal of Physiotherapy.
If you're not a scientist, how can you possibly know if the study is of value?
The more technical the research, the harder it is to understand. For those of us who are not scientists, we have to rely on the source of the information as well as read the details of the study carefully.
Unfortunately, the news media may be your worse source — or the company who's trying to sell you a product. As with everything, you have to be a savvy consumer.
There are some websites where you can find medical studies discussed in laymen's terms and the source is reliable. One is simply named Cochrane which is mostly about medicine and health. From their homepage you can go to the Cochrane Library and read about older studies.
If you want to see a humourous but serious look at this topic, watch this video with John Oliver.
Studies have to have good science behind them.
If you see a headline like this — "Study Shows that Eating Chocolate Every Day will Help you Lose Weight" — it's probably worth checking that carefully. A great many of the "study reveals" stories on the evening news look for a good headline, but there's not always good science behind them. Before allowing yourself to believe it — and perhaps even buy the product or change your eating or fitness habits — be sure to check its authenticity.
I am a BCRPA-certified fitness instructor in Vancouver, BC. I teach four classes at the West End Community Centre in Vancouver, BC, mostly designed for the older adult. The Inevitable Disclaimer: Everything published here expresses only my opinion, based on my training and research. What you do with the information is entirely your own responsibility. I am not liable for any injury you suffer that seems to be related to anything you read here. Always consult your doctor before beginning an exercise program. For other articles, return to the table of contents.