experimental control analysis

The Scholarly Kitchen

What’s Hot and Cooking In Scholarly Publishing

Understanding Experimental Controls

• Experimentation

Much of the training that scientists receive in graduate school is experiential, you learn how to do an experiment by working in a laboratory and performing experiments. In my opinion, not enough time and effort is devoted to understanding the philosophy and methods of experimental design.

An experiment without the proper controls is meaningless. Controls allow the experimenter to minimize the effects of factors other than the one being tested. It’s how we know an experiment is testing the thing it claims to be testing.

This goes beyond science — controls are necessary for any sort of experimental testing, no matter the subject area. This is often why so many bibliometric studies of the research literature are so problematic. Inadequate controls are often performed which fail to eliminate the effects of confounding factors, leaving the causality of any effect seen to be undetermined.

Novartis’ David Glass has put together the videos below, showing some of the basics of experimental validation and controls (Full disclosure: I was an editor on the first edition of David’s book on experimental design). These short videos offer quick lessons in positive and negative controls, as well as how to validate your experimental system.

These are great starting points, and I highly recommend Glass’ book, now in its second edition , if you want to dig deeper and understand the nuances of the different types of negative and positive controls, not to mention method and reagent controls, subject controls, assumption controls and experimentalist controls.

David Crotty

David Crotty is a Senior Consultant at Clarke & Esposito, a boutique management consulting firm focused on strategic issues related to professional and academic publishing and information services. Previously, David was the Editorial Director, Journals Policy for Oxford University Press. He oversaw journal policy across OUP’s journals program, drove technological innovation, and served as an information officer. David acquired and managed a suite of research society-owned journals with OUP, and before that was the Executive Editor for Cold Spring Harbor Laboratory Press, where he created and edited new science books and journals, along with serving as a journal Editor-in-Chief. He has served on the Board of Directors for the STM Association, the Society for Scholarly Publishing and CHOR, Inc., as well as The AAP-PSP Executive Council. David received his PhD in Genetics from Columbia University and did developmental neuroscience research at Caltech before moving from the bench to publishing.

7 Thoughts on "Understanding Experimental Controls"

We could add one more necessary control in this experiment–controlling for variability in individual response.

In the three videos, the experimenter may only detect differences between groups (or average differences). He is unable to detect changes in individuals. Some participants may be more sensitive to caffeine than others, some may show negative changes, and some may show no changes at all. If we take the blood pressure of participants before they drink coffee, we have a baseline measurement for all individuals. We also have a check on whether the experimenter was able to randomly assign participants to each treatment group.

In effect, each individual is their own control, with a before and after measurement. The experimenter is looking at the change in response of the individual rather than the average effect of the group. It is a much more sensitive way to structure and analyze experiments like this.

• By Phil Davis
• Nov 2, 2018, 8:57 AM

Agreed, these videos only skim the surface (his book goes into much greater detail about a much wider range of controls).

• By David Crotty
• Nov 2, 2018, 9:05 AM

Most experimenters who use random assignment to control and treatment groups have found that post-test only design works as well as pre-/post-test design.

• Nov 2, 2018, 10:01 AM

I don’t see how. By controlling for a potentially large source of variability—the individual participant—statistical tests become much more sensitive to changes than averaging all of that variability by group in a simple post-test design. Second, it is a check to see whether the randomization of participants into groups was successful. In many RTCs in the clinical sciences, there is recruitment bias, allowing for the sicker patients to be placed in the treatment group, for example.

• Nov 2, 2018, 12:55 PM

No mention of Institutional Review Board?! The IRB will raise Dr. Johnson’s own blood pressure.

And then there’s the issue of Dr. Johnson’s White Coat — that might trigger considerable individual variation. (My own blood pressure readings change markedly in the course of a visit to the doctor. )

• Nov 2, 2018, 4:59 PM

I believe that IRB approval is discussed in the video on system validation.

• Nov 2, 2018, 5:02 PM

Late to the debate, but I think those are wonderful. Maybe next Control Kitty will ask just how he assembled all those volunteers for his test to be representative and blinding to minimize bias. Were they self-selected? A bunch of caffeine habituated javaheads who responded to an ad in the coffee shop? I could see another video on randomization and sampling frames. I’m sure David Glass’s book goes into all that, but well, I have a shelf full of related books and I’m unlikely to benefit from and want to buy another. Unless maybe he hooks with another clever video or two. Go Kitty! Except, ~900 views! That’s sad. I might have sneak in citations to them. (I tend to get chastised by reviewers/editors for citing non-scholarly sources.) Something like this might slip under the editor’s radar: Glass, D. 2018. Experimental Design for Biologists: 1. System Validation. Video (4:06 minutes). YouTube. https://www.youtube.com/watch?v=qK9fXYDs–8 [Accessed November 11, 2018].

• By Chris Mebane
• Nov 12, 2018, 12:17 AM

Comments are closed.

Related Articles:

Next Article: