Reproducibility: Or Not?

A study that demonstrated a disappointing lack of reproducibility in studies in psychology has recently received much publicity in scientific journals as well as popular media, including the New York Times. Without going into the details of the study, only 39% of the effect sizes in 100 articles could be reproduced. In other words, an admittedly overly simple interpretation is that 39% of the findings in psychological science journals could be reproduced [citation is at the end of this post]

There are all sorts of potential explanations. However, one of the tenets of scientific research is that the findings in a study should be reproducible (or replicable). If this is not the case, then the validity of the study is called into question. Putatively, sometimes something as simple as an error in data entry or faulty use of a statistical package such as STATA, SAS, or SPSS could be at fault.

How much research in scientific psychology is in fact “true?” More generally, how much research in medical, epidemiologic, and social science research is reproducible? In clinical studies, the ability to replicate findings using the actual original data is still difficult, but it is becoming easier because Congressionally mandated NIH funded data collection needs to be shared. A data sharing plan is mandatory in NIH proposals. However, replicability is all too rare and cumbersome.

The public is frequently confused when apparently conflicting results in clinical studies are publicized in the press. There are all sorts of examples, ranging from the relative contributions of diet and exercise in prevention of cardiovascular disease; the relative benefits of medication and therapy in treating affective disorders such as depression; and whether or not vitamin D supplementation attenuates the risk of diseases ranging from coronary heart disease to multiple sclerosis to affective disorders. Because vitamin D may have some effect on immune modulation, some studies have demonstrated that vitamin D deficiency may attenuate the immune response to a variety of pathogens. I am sure that all of you can think of examples.

The most downloaded paper in the PLoS journals, published years before the replicability study in Science is titled “Why most research findings are false”. [citation below] This paper is written mostly about clinical research findings–of fundamental importance in medicine and public health–but it can be extended to many of the empirical biological and social sciences. Some of the reasons for false findings include inadequate study design and statistical power, sampling and study bias, chasing statistical significance rather than a priori specifying the study power and therefore calculating, a priori, the necessary sample size, and low pre and post test probabilities for positive predictive values( aptly called by my colleague Noel Weiss, “the predictive value of a positive test”).

Despite the fundamental need for replication, it is all too difficult to accomplish. My own experience, and the experiences of innumerable colleagues in the social sciences, epidemiology, and clinical sciences is that it is difficult to obtain funding to replicate prior research, and it is equally difficult to publish research that replicates earlier research. I have seen this as a member of many NIH study sections (funding panels), National Academy of Sciences and Institute of Medicine committees, and membership on journal editorial boards. I have experienced this in submitting research papers for publication, and most colleagues with whom I have spoken repeat the same story. A damning critique of research, sometimes regarded as a fatal flaw, is that the research is “derivative” and unoriginal. In other words, it has been done before. This bias makes replication virtually impossible, even though it is fundamentally important in science and social science.

It is equally dismaying that a subpopulation of researchers who are wedded only to open qualitative interviews even deny the importance of replicability and generalizability, stating that the whole point of the effort is not to generalize, but to understand the subjective experiences of perhaps 10-20 research respondents. While I do not deny that there is much to be learned by in depth understanding of subjective experience, a denial of the importance of generalizability is almost proudly publicized as being virtually 0. This will be the subject of a subsequent post.

If 1) much psychological science is not reproducible; 2) much to most research findings are wrong; 3) it is very difficult to obtain funding for “derivative research” or to publish such research, this makes me wonder if our very understanding is misplaced. In clinical epidemiology and in clinical trials, how many of the findings are so false that treatment at the individual levels, or community based public health interventions are fundamentally incorrect? How many medications are administered based upon faulty research, how much surgery is performed out of misplaced trust in the scientific basis for the surgery, and how much public health policy and intervention is based upon findings that cannot be replicated and that may, in fact, be “false”? How much evidence based medicine is based upon incorrect and faulty evidence?

This questioning might be extended to the policy sciences in general. For example, economic and fiscal policies strive to based upon trusted understanding of the factors underlying inflation, unemployment, and economic growth?

In short, how much of our understanding in general is incorrect? Unfortunately that remains unknown.

[Open Science Collaboration, Estimating the reproducibility of psychological science. Science 2015;349:943; DOI: 10.1126/science.aac4716]

[Ioannidis JPA (2005) Why most published
research findings are false. PLoS Med 2(8): e124]

Posted in clinical research, epidemiology, evidence, proof, reproducibility, science | Tagged , , , | Leave a comment

Ebola: No Surprise

Media sources, politicians, and others have treated the current Ebola outbreak in several countries in West Africa as a complete surprise, and therefore, as an unpredictable threat. This “surprise” was not surprising at all. “Unprecedented is a moniker that I seldom saw prior to 9/11. Since then, it has become a cliche. The Ebola epidemic has been described as “unprecedented.” However, it was not unprecedented.

Had it not been Ebola, it would have been Lassa Fever, Marburg, or other hemorrhagic fevers. It could have been many diseases. In fact, the conditions have been ripe for this sort of outbreak for decades–and something like this happened before: SARS. “Emerging infectious diseases” or EIDs have been so notable in infectious diseases and epidemiology that a monthly journal of that title is already in volume 20. It has become one of the most influential journals in the field, with an impact factor of over 7, and has the third highest impact of all infectious disease journals (ISI). The first Institute of Medicine (IOM) report on emerging infectious diseases was published in 1992, and had a tremendous effect on the scientific community, on public health agencies ranging from the local to the WHO, and on policymakers and legislators throughout the world. Laurie Garrett’s well known book, “The Coming Plague,” was published in 1994–twenty years ago. For at least the past 25 years, EIDs have been prominent in my own thinking. The spread of a virus such as Ebola virus was inevitable.

Why the inevitability? Because of social, economic, and demographic changes in societies around the world. Because of economic globalization, and intensifying transportation ties worldwide. The 1992 IOM report identified the “factors in emergence,” and 5 out of the 6 that were identified were social. The sixth, “microbial adaptation and change,” is also social. What the IOM was referring to was the proliferation of antimicrobial resistance, which I will write about in a subsequent blog. The decision to use antimicrobials has a strong social dimension–public expectations of receiving a prescription, lack of public knowledge that these molecules are useless against bacteria, and the spread of antimicrobials to small pharmacies in developing countries, and, indeed, to street markets, where they may be bought freely. And, of course, the use of antibiotics to promote growth in livestock and poultry accounts for 70% of antibiotic use by volume.

Against this backdrop, where viruses and other microbes have existed in micro-foci in isolated portions of rainforest in many tropical areas, and where increasing connectivity with urban areas has constituted a demographic revolution, pathogenic viral and bacterial agents have had a rapidly increasing chance of spreading. It was entirely predictable, and many of us wrote and spoke of this near certainty. Any uncertainty concerned which agent, and where, and when. Predictive epidemiology is not yet at the point where the when and the where could be pinned down.

So was the spread of Ebola a real surprise? Not to those of us in the field. Indeed, not to those of us who saw the movie “Contagion” and realized the scientific basis for that excellent film, which I use as a teaching tool. And not to those of us who realize that the regular and periodic emergence of new influenza strains also represent disease emergence.

The current epidemic was not a surprise. In most ways, it is not even “unprecedented”–a word that I hope to never use in my writing! Not an unpredictable threat: just a threat to health, and, indeed, to economies. Once this outbreak diminishes, and it will, there will be other Ebolas in the future. Because of this, we will need novel public health interventions, and we will need to apply an old intervention on occasion: quarantine, or, the more palatably named “social distancing.” It points to the importance of the public health disciplines to society, and the importance of funding basic and applied epidemiology. I will write more about this issue of funding in the coming weeks.

Posted in Ebola, emerging infectious diseases, epidemiology, Globalization and infectious disease, Health, outbreaks | Tagged | Leave a comment


Wild polio virus has been detected in Israel. Routine environmental/water sampling in southern Israel (Rahat) detected the virus in May. It has now spread to the north. Since the first week of August, the Israeli government has mounted a campaign to administer oral vaccine (OPV) to all children under the age of 10. As of this week, over 180,000 children have been (re)vaccinated.

I must emphasize that no clinical cases of polio have been detected, despite enhanced surveillance. Rather, polio virus has been detected in water samples. That nonetheless poses a risk to any individual with inadequate immunity to the virus: either the non-immunized, or those whose immunity resulting from childhood vaccination has dwindled. And, the troubling question is how virus got into the water supply? This is because there is no known animal reservoir for the virus, and there are likely unsymptomatic carriers in the country.

Heightened wastewater surveillance has detected virus in 67 samples and 24 locations in Israel, according to the Global Polio Eradication Initiative (

It is not unheard to find polio virus isolates in the environment in developed countries with high vaccination rates (Israel’s is 94%). Indeed, there are sporadic reports of polio cases in developed countries. I remember that this was the case right before I left for a conference in Finland in 1985–and I quickly got a polio booster prior to departure. There was a well documented outbreak of clinical cases in the Netherlands in 1992.

A possible scenario explaining this unusual occurrence of polio in Israel is as follows. There are two polio vaccines that are available–the killed virus, injectable vaccine (IPV), and the oral vaccine (OPV), which is a live, attenuated virus vaccine. IPV, which was the standard vaccine given to children in Israel until roughly 1 decade ago, provides good protection to the individual, but still allows people to be adequate carriers of virus–poliovirus is an enteric virus, so the virus, when present, in the gastrointestinal system. The OPV provides mucosal immunity, and decreases shedding of the attenuated virus. To put it another way, IPV provides good protection for the individual, but, to oversimplify, does nothing to prevent passage of the virus from the individual to the environment. OPV, on the other hand, makes this far less likely because of its intestinal immunity. The current vaccination campaign uses only OPV.

The particular strain detected in Israel is identical to a strain detected in Egypt.

There is hope that polio will be eradicated in the near future. The only countries with clinical cases regularly detected are Afghanistan, Pakistan, and Nigeria. Recently, over 100 cases have been detected in Somalia. These are all countries where the public health infrastructure is chaotic, making it difficult to attain high immunization rates.

Posted in Uncategorized | Tagged | Leave a comment

H7N9: Source and origins identified

More of the H7N9 mystery has been solved (or almost solved) in an article published online in Nature today (Wed. Aug. 21, 2013). Let’s remember that H7N9 was first noted earlier this year in China; so far there have been >130 human cases reported, and 40 deaths, for a case fatality ratio of approx. 31%. The public health community–and I–have been concerned about this strain not because of the actual morbidity and mortality, but because of its potential for spread and increase.


Today’s article, published online in Nature (Lam, Wang, Shen et al, The genesis and source of the H7N9 influenza viruses causing human infections in China, doi:10.1038/nature12515) consists of a painstaking process of sampling from the environment as well as oropharyngeal and cloacal swabs from a variety of avian spp., including ducks, geese, and chickens. Phylogenies were then identified using quantitative reverse transciptase PCR (qRT-PCR). Based upon phylogenetic similarity, some of the article’s conclusions are:

1) Transfer of H7 viruses from domestic duck to chicken populations probably occurred at least twice in China;

2) Enzootic H9N2 viruses combined with the H7 viruses to produce H7N9, and a novel H7N7 virus that had not previously been identified. This was in chicken populations, and the novel virus has been shown, experimentally, to infect mammals.

3) Live poultry markets which are found widely in parts of China and other areas of Asia were implicated as potential foci of spread because of the presence of H7N9 in these markets. The authors caution that the H7N9 virus, and H7N7 viruses may be widespread in chickens, and therefore have the possibility of widespread species transfer to humans in these foci.


Echoing so many similar calls and observations, the authors warn that “Long-term influenza surveillance remains essential for early warning of novel reassortant viruses and interspecies transmission.”


This article is the latest of a series of articles on the molecular epidemiology of this novel virus. I argue that a big challenge is to combine two types of analysis: 1) molecular epidemiology identifying “clusters” of similar genomic configurations; and 2) geospatial analysis using GIS, spatial statistics, and spatial modeling, in combination with molecular epidemiology to understand patterns of spread of variants, and, indeed, to serve as early warning and surveillance instruments as new strains of flu and other viruses spread from localized foci, and threaten a more widespread population.

Posted in outbreaks | Tagged | Leave a comment

H5N7: A problem in the making?

A new strain of avian influenza, H5N7, has been in the news recently, and every day brings with it reports of more cases and more deaths, principally in China. I am frequently asked: is this something to worry about?
It depends on the interpretation of the question. Any new strain of influenza is something for concern. Considering the mortality rates and the case fatality ratios of newly reassorted strains of influenza, we could see widespread and significant mortality and morbidity. At the same time, we must be circumspect before declaring a new strain as a potential public health emergency.

Much of the uncertainty comes from the fact that we do not know H5N7’s transmission dynamics: its R(0), or average number of people whom an infected individual infects. We do not know how virulent and pathogenic the strain is in even moderate numbers of people. In short, the basic characteristics of contagion and transmission are unknown. At this point, it is fairly certain that there is no human to human trnsmission. Were there human to human transmission, it might be time to be more concerned. Rather, H5N7 appears to be the product of genetic reassortment involving commercial and domestic fowl, and probably pigs as well. The same complex,, therefore, is probably involved here as it has been with many novel strains.
H5N7 does not appear to be completely new, although the media suggests, in many places, that it is. H5N7 was recognized in Italy in 2007.
So what are we to make out of what we see with H5N7? At this point, it bears watching through various surveillance tools available, including Flu View, WHO, CDC, ProMED, and others. I always have a great deal of respect for the destructive potential of influenza. However, during the same week that CDC has declared our period of seasonal flu transmission to be over, let’s just keep an eye on H5N7.

Posted in Uncategorized | 3 Comments

Chronic pain as a public health problem

Some estimates suggest that 120 million Americans suffer from chronic pain. However, the issue is how to improve access to state of the art pain treatment. It is becoming clearer that treatments that are effective in acute pain, such as opioids, are far less effective in treating chronic pain. Chronic pain is fundamentally different than acute pain since chronic pain involves the remodeling of the brain and CNS. Acute pain is adaptive; chronic pain is usually maladaptive, and serves little physiologic purpose, yet its prevalence is so high.

In the last few years, it has become clear how little we know about public health approaches to chronic pain. Even the epidemiology and risk factors of chronic pain have received little attention–perhaps 10 or 15 people globally have published the major research in pain epidemiology. It is repeated again and again, now, that there is no proof that the use of opioids in chronic pain is appropriate. They may help some people adapt to chronic pain. The ultimate test is whether these individuals are more able to go about activities of daily living, and lives that *they* consider to be happy and productive.

Multiple disciplinary pain research teams have found in the last 50 years that graded exercise is very helpful in most chronic pain states–exercise that is appropriate for the particular individual. This includes both aerobic/cardio exercise, and appropriate muscle training, such as core muscle strengthening exercises. More recently, it has been shown that the use of imagery, relaxation techniques, and mindfulness meditation help to down regulate pain. Movement, exercise, imagery, and mindful breathing all have their places.

In addition, for many types of pain, antiepileptic medication such as gabapentin (Neurontin) and pregabalin (Lyrica, in the US) can help. Even earlier, it became clear that tricyclic antidepressants are helpful. More recently, some of the newer antidepressants, such as venlafaxine (Effexor) and duloxetine (Cymbalta) may be helpful.

Medical intervention aside, might more walkable cities help in the treatment and even prevention of pain? Research has shown that walkable neighborhoods are associated with lower rates of obesity, and possibly lower rates of obesity-related disease, such as diabetes, and cardiovascular disease. Might these walkable neighborhoods–areas with sidewalks, good lighting, and parks, to name a few characteristics–help to prevent and treat chronic pain? A testable hypothesis is that such neighborhoods are associated with lower chronic pain prevalence, after adjusting for confounders and effect modifiers.

The fact that there is no central pain registry, and that it is nearly impossible to get geographically specific data from some of the national health survey data, such as the National Health Interview Survey, hinders pain research. Basically, it is necessary to collect primary data to address these questions, which poses logistical and financial difficulties.

The fact that chronic pain is viewed as a symptom rather than as a set of pathological conditions in themselves may hinder research. This is because most research is disease oriented rather than symptom oriented. For example, most departments of epidemiology have cancer epidemiologists, cardiovascular epidemiologists, neuroepidemiologists, and, going back to the origins of the field, infectious disease epidemiologists. Though there have been several books published on the epidemiology of pain, I know of no epidemiology departments that have pain epidemiologists (maybe I am the only one).

This poses a more fundamental question: is chronic pain a symptom or a disease? A disease in itself that hinders function and happiness. Pain researchers have begun to view chronic pain as a disease. I prefer to consider it as a phenomenon: it just *is*, but chronic pain hinders happiness and harms public health.

If one third of a population (such as in the US) reports that they have chronic pain, we are dealing with one of the most prevalent public health problems. It is time to increase funding for and research on chronic pain. It is time to address chronic pain as perhaps the most prevalent public health issues. And, once there is more understanding and more appreciation of chronic pain, it will be time to address it definitively from a public policy perspective.

Posted in Pain | Tagged | Leave a comment

Flu season 2012-2013 could be bad

The CDC influenza surveillance system indicates that seasonal flu (H3N2) has gotten off to an earlier start than usual, and the attack rates are higher than usual. Collaborating labs are reporting steadily increasing numbers of isolates. There is only one reported case of a novel virus, indicating that genetic shift has not occurred. The ILI (influenza like illness) surveillance system indicates very high activity in Texas and in some of the southeastern states. See:

for CDC’s “Fluview”. 

The actual trends have been presaged by similar increases in search term inquiries in Google Flu Trends, which has been shown to predict actual flu activity–and precede surveillance data by weeks.

Posted in Uncategorized | Tagged | Leave a comment