What Has Happened To The Medical Professions Ability To Apply The Scientific Method To Our Medical Articles?
Stanley Feld M.D.,FACP,MACE
This entry might be a little over the top for many of my readers. However, it is an important entry for our understanding of how the medical profession is contributing to the dysfunction of the healthcare system.
There is much skepticism about the medical profession. I have avoided discussing my professions problems so far. The profession has created many of its problems. Our own colleagues have undermined our own credibility with the publication of flawed scientific articles that have resulted in uncertainty about medical treatments. This subversion decreases patients’ confidence in the medical community’s ability to treat patients’ problems effectively. Public sensationalizing by the media of the contradictory results of clinical research has undermined the patient physician relationship. The re-evaluation of clinical research results has led to contradictory conclusions since the beginning of modern medicine. The constant testing of results and hypothesis is a good thing. We have always been searching for the truth through more objective scientific information.
It seems to me that recently there has been a loose use of statistics in interpreting clinical research, leading to inaccurate conclusions. These conclusions are broadcast and publicized by the media as truth before the conclusions are carefully evaluated by the medical community. In our sound bite society the conclusion counts and not the facts.
I have concentrated on the problems the facilitator stakeholders impose on effective medical care in a dysfunctional healthcare system. It is appropriate to point out some of our intra-professional contributions to the dysfunction of the healthcare system.
Steven Nissen M.D. and Kath Wolski M.P.H. of Cleveland Clinic published an article, “The Effect of Rosiglitazone On The Risk Of Myocardial Infarction And Death From Cardiovascular Causes.” The New England Journal of Medicine published the article. The publication of this article, in my view, contributes to the dysfunction of our healthcare system. The article misrepresents evidence based medicine. Evidence based medicine has become the holy grail of medical practice. Evidence based medicine is gleaned from the published medical literature. It includes commentary and review of the evidence by experts in each field.
According to the Centre for Evidence-Based Medicine, “Evidence-based medicine is the conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients.” The operative words are “use of current best evidence.”
There are three problems with Dr. Nissen’s paper. The design of the study is flawed. The collection of data is incorrect. The results derived from defective data are not statistically significant.
Marshell McLuhan pointed out that “The Media Is The Message” in his landmark book of 1967.
The NEJM has been criticized in the past for pre-releasing information to the press, before the medical community has a chance to evaluate the quality of the information. The medical literature judges have also made mistakes in evaluating data in the past. In my opinion, they have once again made a mistake with Dr. Nissen’s article. The public prejudgments are developed by media stories. The media broadcasts the results (sound bite) and not the facts. The media is the message!
The media loves to expose the deficiencies of the medical profession, the healthcare system and the Federal Drug Administration. Every federal agency is hobbled by bureaucracy. We all realize it. It is one of the reasons will mistrust putting the responsibility for our healthcare in the hands of government. However, sometimes the FDA is criticized unfairly. The unfair criticism receives a lot of media coverage even if the facts are incorrect.
Dr. Nissen and others have criticized the FDA for its inadequacy of evaluating new drugs in general and rosiglitazone in particular because of the results of his study. “The FDA considers demonstration of a sustained reduction in blood glucose levels with an acceptable safety profile adequate for approval of antidiabetic agents. However, the ultimate value of antidiabetic therapy is the reduction of the complications of diabetes, not improvement in a laboratory measure of glycemic control. After the failure of muraglitazar and the apparent increase in adverse cardiovascular outcomes with rosiglitazone, the use of blood glucose measurements as a surrogate end point in regulatory approval must be carefully reexamined,” Dr. Nissan writes.
I agree with Dr. Nissen. A surrogate measurement of a drugs effectiveness is not as precise as the measuring the direct outcome of namely decreasing the complications of Diabetes Mellitus. However, in the DCCT study of 1993the surrogate measurement of blood sugar control (HbA1c) has clearly demonstrated a reduction in the complication of diabetes mellitus. It is totally acceptable to use this information in subsequent studies.
Dr Nissen’s view about the FDA is shared by Psaty and Furberg, who write: “Ongoing trials using rosiglitazone may provide important new data, but for a drug approved in 1999, the delay in obtaining information about health outcomes has already been considerable.” They add that tens of millions of prescriptions for rosiglitazone have been written, and if the current findings represent a valid estimate of the risk of cardiovascular events, rosiglitazone represents a “major failure of the drug-use and drug-approval process in the United States.”
I do not believe that Dr. Nissen’s findings are an estimate of the risk of cardiovascular events with the use of rosiglitazone. Perhaps it is not a failure in the drug-use and drug approval process used by the FDA. It is in the lack of validity of the results of Dr. Nissans’ paper that should be criticized.
Dr. Steven Nissen’s recent paper promotes the distrust of the medical profession and its regulators. The medical profession did not have a chance to evaluate this evidence before it was prejudged in the media. Dr. Nissen has been on network and public television and radio multiple times. Sensationalism in clinical research only serves to decrease the confidence of the public for the value of clinical research.
1. The design of the study is flawed
The study subjects were derived from a meta-analysis. A meta-analysis is an analysis that combines all studies on a subject into one study. The goal is to derive a larger population than each individual study. The purpose of a meta-analysis is to see if an adverse event that is statistically significant is occurring that was not apparent in smaller studies.
The concept of meta-analysis was introduced to clinical medical research in 1992. Most of us never understood the value of meta-analysis in medicine. The design of the various studies combined usually do not have matching protocols. Dr. Nissen’s study combined studies with varied protocols. In fact, similar studies that had no adverse effects either from the placebo or treatment side were eliminated from his meta-analysis. The inclusion of these studies in the meta- analysis might have diluted the effect he was seeking making his results less statistically significant. His data should be reevaluated including these data.
2. The collection of data is incorrect.
Dr. Nissen pointed out the shortcomings of the meta-analysis.” They point out that this meta-analysis is limited by a lack of access to original source data, which would have enabled time-to-event analysis, and on a relatively small number of events (there were 86 MIs and 39 cardiovascular deaths in the rosiglitazone patients vs 72 MIs and 22 cardiovascular deaths in control patients). But they say that despite these limitations, patients and providers should consider the potential for serious adverse cardiac effects of treatment with rosiglitazone.”
I agree, but by his own statistical analysis cardiovascular deaths were not significant and myocardial infarction results were barely significant.
3. The results derived from defective data are not statistically significant.
In order for a result to be statistically significant the confidence interval should not cross one(1). In the analysis of cardiovascular death the confidence interval crossed 1. (C.I. 0.98-2.74). A p value of .06 is not significant. p value should be no greater than 0.05
The C.I. from the analysis of myocardial infarction almost touched 1. (C.I. 1.03-1.98). A p value of .03 is barely significant. Many statisticians believe a significant odd ratio must exceed 2 to be significant. The odds ratio for cardiovascular death was 1.64 and for myocardial infarction 1.43. Those odds ratio numbers are strange because the non significant finding of cardiovascular deaths odds ratio is closer to 2 and a greater trend toward significance, than the barely significant findings of myocardial infarction.
If Dr. Nissen did not remove the clinical trials that did not produce adverse effects in his meta-analysis, the total number of patients in his analysis would be increased and would probably have changed the myocardial infarction confidence interval so that it would cross 1 and p value above 0.05 making the result non significant.
All of the patients in the studies retained were poorly controlled diabetics with HbA1c levels above 8%. Normal should be 6% or below. A high HbA1c alone would increase the incidence of myocardial infarction and cardiovascular death. There is no indication of the distribution of the HbA1c control in the multiple populations. A criticism is that there are two many confounding variables in this studies that can result in an increase in myocardial infarction and cardiovascular death. These confounding variable are not examined independently among the studies. Focusing on one end point, the effect of rosiglitasone on myocardial infarction and death from myocardial infarction without consideration for varience of the other variables is unacceptable.
Rosiglitazone might cause an increase in myocardial infarction and cardiovascular death. However, Dr. Nissen’s paper did not demonstrate this relationship. The public and patients on rosiglitazone have be confused by being presented with the results without rigorous scientific critique of the study.
The confidence of the public in medicine has been shaken by the method of presentation of the results. There have been other effective medications that have suffered the same consequences. The consequences of prejudgment of the results without scientific comment have deprived the public of effective treatments in the past.
In my view this decreases the confidence the public has in the medical care system and adds to the dysfunction of the healthcare system.