Fact or Fiction_ Interpreting Research Articles

Fact or Fiction?Interpreting Research Articles
Bill Origer MDOregon Academy of Family PhysiciansApril 12, 2013
Disclaimers
No commercial financial supportI buy my own pens and lunchI’m not an expert on anything, but I read English pretty wellThanks to Ruth Medak MD, Dave Pass MD, Rich Clark MD, George Waldmann MD, Kathy Ketchum RPh who taught me how to do this
How did I get into this??
Medical Director, Samaritan Health Plans 2000-06Wrote & managed formularies for Oregon Health Plan, Medicare & commercial insurance plansUsed by >400 physicians who know where I liveOregon Health Resources Commission 2006-11, Chair, Pharmaceutical Subcommittee ‘08-11Oregon Preferred Drug List Committee 2010Oregon P & T Committee, Chair, 2012-13
No Study is Perfect
Goal: to understand what a study really proves.
Levels of evidence & article type
1A Meta-analysis of randomized controlled trials1B At least one randomized controlled trial2A At least one controlled study without randomization2B At least one type of quasi-experimental studyDescriptive studies, such as comparative studies, correlation studies, or case-controlled studiesExpert committee reports or opinions and/or clinical experience of respected authorities
AlternativeLevels of Evidence
Class 0: Things that I believeClass 0a:Things that I believe despite the dataClass 1: Randomized, controlled clinical trials (RCCTs) that agree with what I believeClass 2: Other prospectively collected dataClass 3: Expert opinionClass 4: RCCTs that don’t agree with what I believeClass 5: What you believe that I don’tThomas P. Bleck MD, Professor of Neurology, University of Virginia BMJ Jan 2000
Background Information
What do we know about the topic?Is study consistent with this?What does this article claim to provide?New informationConfirm tentative informationRehash old stuff, me-too drug
Background Information
Study sponsorshipCommercialIndependentAuthors conflict of interest
Who, What ,How Long, & How Much?
Who– study populationWhat– end pointsWhat– comparison groupHow long– duration of studyHow muchdifferenceStatistical significanceClinical significanceIs it worth it?Cost/risk vs. benefit
Who
What is the study population?Is this the same or different than the population you treat?How do the differences affect understanding the results?
What - End Points
Clinical end points: death, episode of illness (MI, stroke) hospitalizationFunctional end points: employment, disability, school attendanceSurrogate endpoints:Lab value or physical measurementRating questionnaire or symptom scaleHow much difference to your patients?
Dubious End Points
Time for a wound to heal 50%Surgical results 3 months post-op12 week studies of drugs for lifelong illnessAddiction studies: any end point other than total prolonged abstinenceAny rating scale you can’t find in a quick Google search
What – Comparison Group
PlaceboFDA accepts this, even when there are many active treatmentsAnother treatmentIs it the best or standard treatment?Are doses comparable ?RandomizationSources of bias?
How Long?
Duration of study vs. duration of disease or length of usual treatmentLong enough to eliminate random variations in the disease?Long enough to predict a durable response?How long is long enough for lifetime conditions?Beware of studies terminated early - inaccurate
How Much Difference?
Statistical significance means the results are not likely to be from chance.There may be a statistically significant result that does not help the patient.P value <.05Relative risk confidence interval does not include 1.0
How Much Difference?
Clinical significance – will this help your patient with this condition?Will it reduce mortality or complications?Relative risk reduction vs. absolute risk reductionHow does it compare to standard treatments?Efficacy - experimental situationEffectiveness - real worldMost are less effective in the real world
Relative risk reduction vs.Absolute risk reduction
Statin reduces MI risk by 30%!30% of what?
Statin - Annual Rate Reductionfor Stroke or MI
Elevated cholesterol plus:Previous heart attack & diabetes – risk 15-25%Reduction by 5-10% NNT – 10-20Risk factors: diabetes, high blood pressure, smoking, family history - risk 2-5%Reduction by 0.5 – 3% NNT – 33-200No other risk factors: risk 1-3%Reduction by 0 – 0.1% NNT – 1000 or higher
Too Good to be True
Large benefit – 5-fold increase or decrease is uncommon, about 15% of 85,000 trialsMost were the first study of a topic, and usually small, with < 20 subjects3% showed large effects that were reproducible in bigger, well designed studiesRegression to the mean.Stay skeptical.JAMA 10/24/2012
About End Points
Study is designed with power to prove primary end pointIf you have enough end points, some will be positive by chance20 end points, P> 0.05. Odds of one being random?A prioriestablishedbeforetestingPost hocdeterminedafterlooking at the dataSelection bias
Example: ADHD Medications
Stimulant drugs first used in 1935Efficacy of ADHD meds is measured by parent or teacher rating scales of behaviorNo one knows if this correlates with school performance, graduation rates, employment, incarceration rates, etcOnly 3 studies in 75 years have measured effectiveness on an educational outcome
Rating Scales
Sometimes necessary, but easy to abuseAnyone can name, validate, & publish a scalePuts numbers on subjective things – gives an appearance of precision
Rating Scales
Penn State Worry QuestionnaireFecal Incontinence Quality of Life Score (FIQL)Cleveland Clinic Florida Fecal Incontinence Score (CCF-FI)Uterine Fibroid Symptom Quality of Life (UFS-QOL)Aging Male Symptom Score (AMS)Massachusetts General Hospital Hair Pulling ScalePsychiatric Institute TrichotillomaniaScale20-Item Sino-Nasal Outcome Test (SNOT-20)Aberrant Behavior ChecklistVisual Analogue Scale of BothersomenessKing’s health questionnaire version 7 (KHQ) – This is a British questionnaire about urinary incontinence. Which King is not specified.Fred Hutchinson Food Frequency Questionnaire (FFQ)Patient Assessment of Constipation Quality of Life (PAC-QOL)Bristol Stool Form Scale (BSFS)
Origer’s Comments onRating Scales
Many are dubiousIf you can’t find an explanation or a copy of the test easily, it’s not worth muchIf the change is not at least 30% of the scale’s total range it’s probably not significantA statistical difference may not mean anything practical
Do the Math
Calculate the theoretical results for 100 or 1000 patients
Stroke prevention atrial fibPatients unable to take warfarinClopidogrel + ASA vs ASA alone
7554 patients, average age 71Followed 3.6 yearsDecreased stroke rate by 27%Decreased MI rate by 22%Risk of major bleeding 2%Sounds good….until you do the mathNEJM May 14, 2009
Do the Math
Clopidogrel for atrial fib from NEJMTreat 1,000 patients for one yearStroke reduction 3.3% to 2.4% = 0.009 x 1,000 = 9 patientsMI reduction 0.9% to 0.7% = 0.002 x 1,000 =2 patientsPatients w benefit = 11Bleeding increase 1.3% to 2.0% = 0.007x1,000 =7 patientsPatients harmed = 7989 of 1000 patients will not benefit
Do the Math
NNT Number Needed to TreatNNT= number treated ÷ number with benefitNNT= 1000 treated ÷ 11 with benefitNNT = 91You must treat 91 patients for one to benefit90 out of 91 patients will not benefit.98.9% will not benefitDoes this still sound good?
Sample Articles
Background&SponsorshipWho– study populationWhat– end pointsWhat– comparison groupHow long– duration of studyHow muchdifference?Statistical significanceClinical significanceIs it worth it?Cost/risk vs. benefit