Evil clowns Reinhart and Rogoff based their pro-austerity study on a butchered Excel formula
[Adding "... and cherrypicked data!" "Evil clowns" because it's such a grotesque neo-liberal #FAIL. (I mean "fail" in terms of academic and intellectual integrity. Obviously, the paper was a tremendous success in terms of achieving desired policy outcome for its authors and the clients they serviced.) Anyhow, this post started out with one quote from Konczal and turned into a link- and quote-fest ordered by discovery, and ending with the original authors from UMass. Even though the whole topic has gone viral, I'll stick with the festival concept, so maybe stronger analysts than I am can use this as resource. --lambert]
File under Making Beppe Grillo look good! Mike Konczal:
Coding Error. As Herndon-Ash-Pollin [see below] puts it: "A coding error in the [Reinhart and Rogoff] working spreadsheet [in Growth in a Time of Debt] entirely excludes five countries, Australia, Austria, Belgium, Canada, and Denmark, from the analysis. [Reinhart-Rogoff] averaged cells in lines 30 to 44 instead of lines 30 to 49...This spreadsheet error...is responsible for a -0.3 percentage-point error in RR's published average real GDP growth in the highest public debt/GDP category." Belgium, in particular, has 26 years with debt-to-GDP above 90 percent, with an average growth rate of 2.6 percent (though this is only counted as one total point due to the weighting above).
Being a bit of a doubting Thomas on this coding error, I wouldn't believe unless I touched the digital Excel wound myself. One of the authors was able to show me that, and here it is. You can see the Excel blue-box for formulas missing some data:
[Quartz has a deadpan post on how to avoid the Excel error.]
This error is needed to get the results they published, and it would go a long way to explaining why it has been impossible for others to replicate these results. If this error turns out to be an actual mistake Reinhart-Rogoff made, well, all I can hope is that future historians note that one of the core empirical points providing the intellectual foundation for the global move to austerity in the early 2010s was based on someone accidentally not updating a row formula in Excel.
So what do Herndon-Ash-Pollin conclude? They find "the average real GDP growth rate for countries carrying a public debt-to-GDP ratio of over 90 percent is actually 2.2 percent, not -0.1 percent as [Reinhart-Rogoff claim]." [UPDATE: To clarify, they find 2.2 percent if they include all the years, weigh by number of years, and avoid the Excel error.] Going further into the data, they are unable to find a breakpoint where growth falls quickly and significantly.
The Reinhart-Rogoff study has been a key ideological prop of pro-austerity polices. Check Reinhart-Rogoff's website for a list of the
dupes economists who cited to it, who will presumably express their deep regrets and embarrassment, and promptly retract have a good laugh together next time they meet in the green room, exactly like the WMD guys did. Of course, austerity has led to the suffering of millions, and the deaths by suicide of many. But austerity was the goal. As along as the bottom line was right, nobody cared about the details. Never, therefore, have I been so happy to claim: "That's not a bug. It's a feature." This story is the WMD story of today's Neo-Depression. (Same post at Seeking Alpha.)
UPDATE Contact information:
Rogoff: Littauer Center 216
Fax: 609-258-5974 E
UPDATE Apparently, I went for the fun part. Oh well!
(I went for the Excel error because Excel errors are so ubiquitous, as for example in Jippy Mo's London Whale trade.)
UPDATE Yglesias says that the original study was so tendentious that nobody will care about an error in it:
This is literally the most influential article cited in public and policy debates about the importance of debt stabilization, so naturally this is going to change everything. Or, rather, it will change nothing. As I've said many times, citations of the Reinhart/Rogoff result in a policy context obviously appealing to a fallacious form of causal inference. There is an overwhelming theoretical argument that slow real growth will lead to a high debt:GDP ratio and thus whether or not you can construct a dataset showing a correlation between the two tells us absolutely nothing about whether high debt loads lead to small growth. The correct causal inference doesn't rule out causation in the direction Reinhart and Rogoff believe in, but the kind of empirical study they've conducted couldn't possibly establish it. To give an example from another domain, you might genuinely wonder if short kids are more likely to end up malnourished because they're not good at fighting for food or something. A study where you conclude that short stature and malnourishment are correlated would give us zero information about this hypothesis, since everyone already knows that malnourishment leads to stunted growth. There might be causation in the other direction as well, but a correlation study woudn't tell you. The fact that Reinhart/Rogoff was widely cited despite its huge obvious theoretical problems leads me to confidently predict that the existence of equally huge, albeit more subtle, empirical problems won't change anything either. As of 2007 there was a widespread belief among elites in the United States and Europe that reductions in retirement benefits were desirable, and subsequent events regarding economic crisis and debt have simply been subsumed into that longstanding view.
Yep. I mean, it's not like we're talking scholarship here.
2) It’s very difficult to know whether the Reinhart & Rogoff finding actually changed the course of policy that was yet to be decided, or if it was simply used as a post-hoc justification for policy that was inevitable anyways. The paper certainly did provide intellectual justification for austerity advocates, but beware of exaggerated claims. It wasn’t the only paper to arrive at such a conclusion, and the R & R study had come under plenty of previous criticism.
Still, a case to consider — Olli Rehn’s citation of “serious economic research” (this paper) when defending “fiscal consolidation” in Europe this year, including the 90 per cent debt figure. Rehn’s stance was widely criticised in some detail at the time, of course. But we have to admit, it’s very easy for a politician to answer back with a single figure like this, whether or not the figure originally wagged the dog and inspired policy in the first place. And it’s problematic when public officials fail to appreciate the extent to which papers like these need vetting.
On "used as a post-hoc justification for policy," the paper clearly was. That doesn't absolve Reinhart or Rogoff of the scholar's ethical responsibility not to propagate error. On "wasn’t the only paper," normally I hate prostitution metaphors in political economy (they are unfair to prostitutes), but yes, in this case Reinhart and Rogoff weren't the only ladies of negotiable affection in the brothel that is neo-liberal economics. But they were and are what they are, despite the undeniable fact there are many others like them, aching to be them.
UPDATE Economist's View:
The work of Reinhart and Rogoff was a major reason for the push for austerity at a time when expansionary policy was called for, i.e. their work supported the bad idea that austerity during a recession can actually be stimulative. It isn't as the events in Europe have shown conclusively. To be fair, as I discussed here (in "Austerity Can Wait for Sunnier Days") after watching Reinhart give a talk on this topic at an INET conference, she didn't assert that contractionary policy was somehow expansionary (i.e. she did not claim the confidence fairy would more than offset the negative short-run effects of austerity). What she asserted is that pain now -- austerity -- can avoid even more pain down the road in the form of lower economic growth. Here's the problem. She is right that austerity causes pain in the short-run. But according to a review of her work with Rogoff discussed below, the lower growth from debt levels above 90 percent that austerity is supposed to avoid turns out, it appears, to be largely the result of errors in the research. In fact, there is no substantial growth penalty from high debt levels, and hence not much gain from short-run austerity.
UPDATE Dean Baker:
This is a big deal because politicians around the world have used this finding from R&R to justify austerity measures that have slowed growth and raised unemployment. In the United States many politicians have pointed to R&R's work as justification for deficit reduction even though the economy is far below full employment by any reasonable measure. In Europe, R&R's work and its derivatives have been used to justify austerity policies that have pushed the unemployment rate over 10 percent for the euro zone as a whole and above 20 percent in Greece and Spain. In other words, this is a mistake that has had enormous consequences. In fairness, there has been other research that makes similar claims, including more recent work by Reinhardt and Rogoff. But it was the initial R&R papers that created the framework for most of the subsequent policy debate. And [Herndon, Ash, and Pollin have] shown that the key finding that debt slows growth was driven overwhelmingly by the exclusion of 4 years of data from New Zealand.
UPDATE And here is ground zero: Herndon, Ash, and Pollin from UMass's Political Economy Research Institute:
Herndon, Ash and Pollin replicate Reinhart and Rogoff and find that coding errors, selective exclusion of available data, and unconventional weighting of summary statistics lead to serious errors that inaccurately represent the relationship between public debt and GDP growth among 20 advanced economies in the post-war period. They find that when properly calculated, the average real GDP growth rate for countries carrying a public-debt-to-GDP ratio of over 90 percent is actually 2.2 percent, not -0:1 percent as published in Reinhart and Rogoff. That is, contrary to RR, average GDP growth at public debt/GDP ratios over 90 percent is not dramatically different than when debt/GDP ratios are lower. The authors also show how the relationship between public debt and GDP growth varies significantly by time period and country. Overall, the evidence we review contradicts Reinhart and Rogoff's claim to have identified an important stylized fact, that public debt loads greater than 90 percent of GDP consistently reduce GDP growth.
Funny how scholars at "second rank" schools like UMass and UMKC are beating the shills and whores at Yale and Princeton like gongs.
Holy Coding Error, Batman [T]he really guilty parties here are all the people who seized on a disputed research result, knowing nothing about the research, because it said what they wanted to hear.
UPDATE R-R respond. Shorter: Bad data and coding errors don't matter because others got the same results. Because scholarship! (Oh, and the first word in the response? "Ben." That would be Ben Smith of Politico, which shows you where this is going to be played out, for good or ill.
UPDATE Yglesias responds to R-R:
Carmen Reinhart and Vincent Rogoff have emailed a response to today's criticisms of their research on debt and growth and I've pasted the full text of their response below the fold. As I predicted, the criticisms change nothing. To sidestep this debate for a moment, I would note [Ugh. Beltway subjunctive] in particular that the entire debate completely fails to acknowledge the main flaw in the R&R research. As they themselves acknowledge in the response, the empirical correlation that we're arguing about is completely irrelevant to the policy debate at hand. They write that "we are very careful in all our papers to speak of 'association' and not 'causality'". This genuinely ought to settle the debate. Nothing Reinhart and Rogoff present, under any interpretation or any methodology offers any reason to believe that a high debt:GDP ratio causes slow growth. Yet much political rhetoric, including some from Reinhart and Rogoff themselves, presents their work as offering important policy guidance. It's great that when challenged they retreat to the more defensible claim that their work is actually irrelevant [Ouch!], but many policymakers and pundits seem to feel otherwise.
Ha ha! Say, where's Young Ezra on this? Waiting to weigh in and settle things?
UPDATE Which party can blame the other first?! From The Hill:
[R-R's] work has been cited not just by Republicans, but members of both parties training their fire on the nation's deficit. Former Senate Budget Committee Chairman Kent Conrad (D-N.D.) cited the study on the Senate floor in his farewell address before his retirement in 2012. Calling it a "landmark work," he used it to underscore his persistent push to shrink the nation's debt.
Now I need to find an Obama quote on debt-to-GDP ratio to go along with Conrad's.... Checking back in for another
UPDATE, Ars Technica points to the lack of peer review:
The paper's problems also highlight the value of peer review. Although the paper was included in the American Economic Review, it was included as a "Papers and Proceedings" article, taken from a conference and published without being reviewed. A peer reviewer might have sought the Excel spreadsheet at review time and could have exposed these flaws sooner. In turn, this may have prevented an apparently flawed paper from receiving the widespread attention that it got. The response is similarly unreviewed, so it too may be flawed.
Well, that's neo-feudalism for you! And actually, the response is reviewed: The spreadsheet is there for all to test. And it contains the errors cited. One does wonder whether the "editors" of the American Economic Review will issue a statement. "Mistakes were made," perhaps?
UPDATE Bloomberg lets the UMass authors put the "political" in "political economy," right where it belongs:
The findings by Reinhart and Rogoff “have served as an intellectual bulwark in support of austerity policies,” the University of Massachusetts academics said. The fact that their results “are wrong should therefore lead us to reassess the austerity agenda.”
Well, only if evidence is a concern. Let's be careful to caveat that.
UPDATE George Osborne, 2010 (via The Telegraph):
[OSBORNE] As Rogoff and Reinhart demonstrate convincingly, all financial crises ultimately have their origins in one thing – rapid and unsustainable increases in debt. As they write, "if there is one common theme… it is that excessive debt accumulation, whether it be by the government, banks, corporations, or consumers, often poses greater systemic risks that it seems during a boom.
And the Telegraph comments:
So is this merely an academic point, an obscure intellectual exercise with no political meaning? Not quite. If – and it remains an “if” – Rogoff and Reinhard are discredited, Mr Osborne has lost one of the intellectual props for his political-economic judgement. In essence, his position has become just a little more exposed. If the story of this government is the story of George Osborne’s era-defining economic gamble, then the stakes have just got even higher.
UPDATE Quartz on policy-maker buy-in. Not just Osborne!
Were other policymakers around the world paying attention? Absolutely: “[I]t is widely acknowledged, based on serious research, that when public debt levels rise about 90% they tend to have a negative economic dynamism, which translates into low growth for many years.” — European Commissioner Olli Rehn. “Economists who have studied sovereign debt tell us that letting total debt rise above 90 percent of GDP creates a drag on economic growth and intensifies the risk of a debt-fueled economic crisis.” — House Budget Committee Chairman and former Republican vice-presidential candidate Paul Ryan. “It’s an excellent study, although in some ways what you’ve summarized understates the risks.”— Former US Treasury Secretary Tim Geithner. [If Geithner, then Obama, but I'd want to be 100% certain Geithner's talking about the 2010 paper, not the 2009 book. A Google search doesn't turn up the original sourcing.] “[W]e would soon get to a situation in which a debt-to-GDP ratio would be 100%. As economists such as Reinhart and Rogoff have argued, that is the level at which the overall stock of debt becomes dangerous for the long-term growth of an economy. They would argue that that is why Japan has had such a bad time for such a long period. If deficits really solved long-term economic growth, Japan would not have been stranded in the situation in which it has been for such a long time.” Lord Lamont of Lerwick, former UK chancellor and sometime adviser to current chancellor George Osborne. “The debt hurts the economy already. The canonical work of Carmen Reinhart and Kenneth Rogoff and its successors carry a clear message: countries that have gross government debt in excess of 90% of Gross Domestic Product (GDP) are in the debt danger zone. Entering the zone means slower economic growth.”— Doug Holtz-Eakin, Chairman of the American Action Forum.
UPDATE Economix's Anne Lowry:
Flaws Are Cited in a Landmark Study on Debt and Growth Moreover, the argument floating around that the Reinhart-Rogoff research altered any country’s policy course – the idea that we have austerity because of a spreadsheet mistake – is dubious at best. When the global recession hit, many members of the policy elites in Europe and in the United States already believed that budget discipline begat growth. And countries that adopted austerity budgets often had to do so because of the vicissitudes of the market and the financing constraints of groups like the International Monetary Fund and the European Commission.
In short, no harm, no foul. The Very Serious People agree, and what's a little bad data and a few coding errors between friends?
UPDATE Daniel Akst at the WSJ is all "he said, she said:
Is 90% Really a Sovereign-Debt Red Line? A New Paper Says No Who’s right? You be the judge. Meanwhile, it’s unlikely we’ve heard the last of this.
Well, yes. Now you can be the judge, because the UMass people posted the data and the spreadsheet. What does that tell you?
UPDATE Reuters quotes the RR response:
"Of course much further research is needed as the data we developed and is being used in these studies is new," they said in a joint response by email. "Nevertheless, the weight of the evidence to date - including this latest comment - seems entirely consistent with our original interpretation of the data."
Well, copy edit problems here; "more" research isn't needed; research is needed. Bad data and bad coding aren't research (unless you confuse research with funding, of course). The question somebody should ask RR is this: Will all the data and the spreadsheets for their 2009 book now be made available? We need to know if the 2010 ("Growth in a Time of Debt") paper is an isolated incident -- sloppy work from a
fall guy grad student, perhaps? -- or whether the methodological, er, issues with the 2010 paper are pervasive through their oeuvre. After all, fakers fake; they don't do it once and then stop.
Who among us hasn't made an Excel formula error ... that became the intellectual vindication for self-defeating economic policies?
— Matt O'Brien (@ObsoleteDogma) April 16, 2013
Zeitgeist alert! First "The TA is to blame!" sighting:
@obsoletedogma seriously, I guarantee you it was an undergraduate RA coding error and the author didn't catch it. Been there.
— Knows (@davidsonangmoh) April 16, 2013
UPDATE LA Times's Michael Hiltzik responds to RR's response:
In their response, R & R don't directly address [yep] the UMass critique that their math was erroneous. Instead, they argue that since the critique also finds lower economic growth in high-debt situations, the new findings are no big deal. The point, however, is that the corrected figures don't show anything like a break point in economic growth at the 90% ratio, which Rogoff and Reinhart cited.
R & R also point readers to a 2012 paper that they say validated their original findings. That paper, however, is more nebulous than they suggest. In it they found lower growth in high-debt-ratio conditions, but it also acknowledged that the high debt was associated with other growth-reducing conditions, including wars, banking crises and depressions. If that sounds like the developed world in 2008-2013, no kidding.
That underscores what has been one of the most frequent critiques of the original paper: Even if high debt/GDP ratios are associated with low growth, what's the evidence that the former causes the latter? [If RR have any, they'd better make the data and the spreadsheet public, eh?] It's been widely pointed out that it's just as likely, even more likely, that low growth drives higher government debt, rather than the other way around.
One reason is that lower growth will suppress GDP, thus directly affecting the mathematical ratio. Another is that low growth spurs governments to take on more debt as a recovery tool. Under both rationales, it makes sense to respond to economic slumps with more government spending, not with austerity. That's where fans of Rogoff and Reinhart, like Rep. Paul Ryan, have gotten things exactly wrong.
UPDATE Matt Bruenig at TAP dances on the grave:
For those in policy circles, it is hard to understate how big a deal this actually is. Pro-austerity forces obviously have separate motives for why they want to bring on the economic pain that has nothing to do with Reinhart and Rogoff. But for the last few years, this research has been providing the pro-austerity crowd intellectual cover for their policy preferences. Everywhere you turn in these debates, you are bombarded with Reinhart and Rogoff. While there were plenty of good reasons to dismiss their theory before now, the fact that their own data—when corrected for basic coding—cuts against them should officially render this theory dead. It is simply not the case that a government with a debt-to-GDP ratio over 90 percent is a death sentence for economic growth. Reinhart and Rogoff were wrong even on their own terms.
The theory will be dead when Obama kills it. Or rather, the theory will no longer be undead when Obama destroys its brain.