With the recent talk of p-values and their value in scientific journals it brings to light an important interpretive tool in efficacy of therapies, clinical experience. P value is the chance of getting a positive response in a scientific study when there is no real effect after all, also known as a false positive. The smaller this number, the better the certainty that what you are observing is truly an effect of what you are studying. This number often is given as p .05 meaning that only 5% of the time would you see this happen by chance, the rest of the time it is a true effect of what you are studying. Put another way, you can say that you are rejecting the “no effect” assumption, and come to the conclusion that drug A has effect B on the body and claim that your results are statistically significant.

This has been the backbone of science forever to determine if what you are seeing is not a fluke. On closer examination though this value may not be as strong as we first thought. Don’t get me wrong, it is an awesome way to reduce bias in a study and the best we have to weed this out as long as we don’t play around with this p value after our calculations are done. What if we applied this 5% theory to a supplement that was being tested for a certain condition. If we wanted to try 100 supplements for a given condition and only one of these supplements actually did something to improve the condition, we would find 5 supplements that appeared to help (false positives) and one extra that actually did, the one effective supplement in the bunch. Of the six supplements you came away with thinking worked for the condition, really only one worked. This means that out of those six conclusions that claim to help the problem, only 1 in truth really does. You are incorrect 83% of the time in your determination of effective products even though you successfully eliminated 94 ineffective products! Imagine, a randomized, placebo controlled trial with a p value of 0.05 with this kind of result.

Retractions of published papers also appear to be on the rise and after being involved myself this past year in a scientific study, there really is a lot of pressure felt by the authors to get published in a scientific journal. It’s almost like a final approval by the cool kids in class and seems to psychologically give a stamp of approval on your work not only to the authors that did the study, but by the public and scientific community that will read or hear about the study. If you aren’t published, there is almost a sense of failure felt towards the whole project, regardless of how astounding the results are.

This brings us to the world of the front line where these products are actually handed out to the public, the Pharmacy. Many times I see products written on prescription that work exactly the way they are supposed to but sometimes they fail miserably. Regardless of how many studies were done on a drug, if a patient paid $100 for it and it didn’t work, they really don’t care how many studies were done or what the p value was; they are out $100 and they now need to fork over more money for another product. This doesn’t mean the studies that brought this to market were bad, it’s just that they were some of the outliers in the results that didn’t respond to the drug.

When you deal with supplements you often are labeled and dare I say it with “alternative therapy”, you are always searching for these studies. They are often small studies but you still look for them. The same is true for pain compounding. It is not difficult to be labeled a quack or a charlatan when you try to help someone that doesn’t seem to fit into the regular modern medicine model or wants to try another way first. Nothing replaces clinical experience in determination of a product’s net worth and if studies are done correctly your results should mimic the studies you originally read. Keep in mind that this may mean a 70% success rate as determined by the studies. It is only when you see something work before your own eye(s) that makes you comfortable suggesting it more. Those products that showed promise in studies and it doesn’t pan out with your patients, these products fall away rather quickly. When you deal with people that are paying out of pocket for something, you know it is working when they come back for more to spend more money on. I have had physicians steer away from a product because of one or two bad experiences with it with their own patients. As always, patient safety is key with any product. Will this therapy harm this patient based on their existing meds, allergies or medical condition? Will it cause a dangerous delay in treatment with another more proven product? These are important questions to as when a patient looks for an alternative medication.

Clinical experience with pain compounding creams has completely change the thinking of a lot of physicians I deal with at the pharmacy level. Many of these doctors haven’t read even one of the studies I have on the response rate of this type of therapy but when they took a leap of faith with just one patient, then another and another, they realized the value of a therapy they were not taught in school. When I get in my car and turn the key, a lot goes on to start the car and keep it running. I haven’t read any studies on car engines but I do it because it seemed to work for others and it works for me for the most part as well.

False positives and subjective results can happen this way as well, but when a patient that was previously addicted to hydromorphone prefers a pain cream or an addition of omega-3 with their pain medication, it helps to alleviate thoughts that they are pretending the pain went away. As one palliative care physician said to me, “If the placebo effect is 30% on drug X, I’ll take that kind of response rate”. When there are doubts as to the effectiveness of a well-designed trial, clinical experience acts as an effective filter to refine one’s beliefs.