Research Highlights Article
November 16, 2018
Making economics transparent and reproducible
The AEA interviews Ted Miguel about the credibility of economics.
Ted Miguel says that most economists share the core value of transparency, but there are steps the profession can take to encourage more openness.
sqback
The social sciences have a credibility problem, and Berkeley’s Ted Miguel says economics is no exception.
He and co-author Garret Christensen summarize the issues and offer promising solutions in the September issue of the Journal of Economic Literature.
Published research might only show part of the picture, and Miguel says the picture that is shown might be distorted. Journals have a tendency to publish “significant” results -- as opposed to “null” results that show no effect. Economists may also inadvertently bias their results, making it hard to replicate their findings.
But there are reasons to be optimistic. Miguel and Christensen show some steps that scholars are taking to make their research more transparent and reliable. Miguel sees research norms changing quickly and improving among his students.
The AEA spoke with Miguel about the replication problem in economics and how the next generation of researchers is embracing new tools to make the profession more credible.
A transcript edited for length and clarity follows. A full-length audio version of the interview can be heard using the media player below.
AEA: There’s a growing debate about the credibility of the social sciences, what some are calling a “replication crisis.” What kind of challenge do you think the social sciences and economics are facing?
Miguel: There's been increasing evidence really across the social sciences, and economics is no exception, that publication bias is a real concern. The kind of evidence that we see published in our journals may not always be representative of the underlying data and findings but instead findings that have certain characteristics, like very statistically significant findings or findings that are surprising or considered provocative, are maybe more likely to be published than null results or seemingly boring results.
Publication bias is a concern, and then related to this is what you called the replication crisis, which is the concern that findings that are published are not reliable. They can't be reproduced, possibly because of publication bias, possibly because of other researcher practices that make research less credible.
AEA: Why do you think people outside of academia should be worried about this?
Miguel: I think public policymakers, foundations, citizens as a whole look at the published research literature for guidance on many issues. They look to it for guidance on what tax policy should be or what minimum wage policy should be or how we should structure foreign aid programs internationally.
I think that is a real problem because, to the extent that we're only seeing a subset of the research, then it's really unclear how much faith we should have in the published literature.
AEA: What got you interested in thinking about this problem?
Miguel: When I was in graduate school I was part of a group of researchers, faculty and graduate students, that started field experiments, randomized controlled trials (RCT), in development economics. I was drawn to that kind of work because it seemed to be more objective and scientific and rigorous.
As we began to use experiments and sort of systematically design, sort of engage in the design of research projects prospectively through randomized trials, we realized that we could go farther in becoming scientific. So, several years into the use of randomized controlled trials, I remember discussions with colleagues where people said, "Well, another thing that we could do is start writing down our hypotheses and even our statistical code in advance."
I really do see the move toward research transparency and reproducibility as sort of the next step in this push for really more scientific empirical work in economics. Part of that is the use of experimental design when we can use them. Part of it is now being embodied in the use of pre-analysis plans and pre-registration which have really taken off in the last couple of years.
AEA: It seems like the issues you’re raising might be controversial. When you’ve discussed them with your colleagues, have they pushed back? Or do most economists think this is a healthy conversation?
Miguel: I think there has been some controversy; there has been some resistance. But I will say, there hasn't been much conflict in economics about these issues relative to other fields. I think there's been some debate, some discussion, some resistance, some disagreement, but the general trend towards openness, transparency is really strong in economics.
In five years we're going to look back. I think it's already happening. My students say, "Gosh you know when you were in grad school and no one shared their data, no one poked through their data, how could that be?" But that's just how it was. And I think for development students now, my PhD students at Berkeley, they write pre-analysis plans. That's just part of their work flow. And I think five or ten years from now they are going to say, "I can't believe people used to not do this. How could you not write down your hypotheses before you run a million dollar experiment? How can you not write down your hypotheses?"
The feeling is it's the kiss of death for publication if you get a null result.
-Ted Miguel
AEA: What’s the one problem you think we really need to tackle?
Miguel: I do think that an absolute, fundamental improvement which has already occurred to a large degree has been sharing of data and statistical code. The American Economic Review was really in the lead in sort of pushing for data sharing about ten years ago. I think that was really a fundamental change.
The concern around that is that more and more people are using proprietary data. So the trend towards open data and data sharing, which has been so strong, is sort of stalling out a little bit right now. And it's not clear exactly how to solve that problem, because very often the people who share data with researchers don't want it to be public. It's going to be incredibly important over the next five or ten years to allow people to access data without sort of holding the data.
♦
“Transparency, Reproducibility, and the Credibility of Economics Research” appears in the September issue of the Journal of Economic Literature. Music in the audio is by Podington Bear.