Looking for the rabbit in the field: my study disproves your study.

IMG_2313[2]

When I am asked to “prove” something by showing a study, it reminds me of how backwards we can be when it comes to stacking studies against each other to make a point.  We come up with rebuttals like, “the patient size was too small”,  “the study is too old” and “the study wasn’t double blinded”.  In fact, virtually every study that has ever been done can be debated for some reason or other.  As well, it is safe to say that not much has ever been proven of any value or consequence by just one study alone.

Let’s say that a colleague and myself have a disagreement on whether or not chromium has the ability to prevent hypoglycemia.  This can start out with a quick exchange of some handpicked positive and negative outcome studies that may be well designed or not or maybe had the power to determine the outcome or not.  Maybe a study hid some of the data or the statistical analysis was done improperly.  Maybe my colleague had 10 times as many studies compared to mine.  Is that what decides the outcome of all of these scientific studies?  That one person can scrounge up more studies than someone else?  How does that person explain a well-designed study that doesn’t back up his or her argument?  Do they quietly pass off the study as a fluke or placebo effect or do they start to see what scientific study is for us – many studies that create a whole picture much like individual pieces of a puzzle make a picture.

Let’s say we have a puzzle.  It is a puzzle of a field of green grass on a sunny day with blue sky and some white clouds.  It turns out that this picture is a picture of a rabbit sitting in the field, only a very small part of the picture, less than one percent is taken up with the rabbit.  This translates into just 2 pieces of the 500 piece puzzle.  Let’s say my colleague and I divide the puzzle in two and each take half of the puzzle home with us.  We separately lay out the pieces and get a vague idea about what the picture shows.  It turns out I have both pieces that make up the rabbit, statistically not what we would expect but it happened.  I call my colleague and tell him about the picture we are looking at and how it appears to be one with a rabbit in the field.  He says, “that’s crazy, there’s no rabbit in this picture”.  He said he’s looked at all of his pieces and no evidence of any rabbit.

So who is correct?  Well based on what is in front of each of us, we are both correct.  When we both meet to put the puzzle together on one table, the entire picture becomes clear.  The puzzle changes from a field on a sunny day to a rabbit on a sunny day.  Each piece of the puzzle is a scientific study.

This exercise shows us what the power of a study means, or an analysis of many studies can show.  For an infrequent result, the amount of pieces that are required to find the rabbit are large.  My colleague arrived at what would be called a negative result and mine was a positive result.  In a book called Statistics Done Wrong, (Alex Reinhart) it is described that in a review of studies between 1975 and 1990 in prestigious medical journals, almost a third of these studies yielded negative results.  A full 64% of these studies didn’t have the power to determine a 50% difference in the primary outcome they were looking for between treatment groups.  When a 25% difference was present, a full 84% of studies didn’t have the power to find this.

As always, I believe that scientific studies are the best thing we have to unlock what we do not know.   One study however, can be deceiving when taken as a single data point on a graph.  Scientific studies are a group effort.

 


Posted

in

by

Tags:

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *