If you have "no place to go," come here!

Evil clowns Reinhart and Rogoff based their pro-austerity study on a butchered Excel formula

[Adding "... and cherrypicked data!" "Evil clowns" because it's such a grotesque neo-liberal #FAIL. (I mean "fail" in terms of academic and intellectual integrity. Obviously, the paper was a tremendous success in terms of achieving desired policy outcome for its authors and the clients they serviced.) Anyhow, this post started out with one quote from Konczal and turned into a link- and quote-fest ordered by discovery, and ending with the original authors from UMass. Even though the whole topic has gone viral, I'll stick with the festival concept, so maybe stronger analysts than I am can use this as resource. --lambert]

File under Making Beppe Grillo look good! Mike Konczal:

Coding Error. As Herndon-Ash-Pollin [see below] puts it: "A coding error in the [Reinhart and Rogoff] working spreadsheet [in Growth in a Time of Debt] entirely excludes five countries, Australia, Austria, Belgium, Canada, and Denmark, from the analysis. [Reinhart-Rogoff] averaged cells in lines 30 to 44 instead of lines 30 to 49...This spreadsheet responsible for a -0.3 percentage-point error in RR's published average real GDP growth in the highest public debt/GDP category." Belgium, in particular, has 26 years with debt-to-GDP above 90 percent, with an average growth rate of 2.6 percent (though this is only counted as one total point due to the weighting above).


Being a bit of a doubting Thomas on this coding error, I wouldn't believe unless I touched the digital Excel wound myself. One of the authors was able to show me that, and here it is. You can see the Excel blue-box for formulas missing some data:

[Quartz has a deadpan post on how to avoid the Excel error.]

This error is needed to get the results they published, and it would go a long way to explaining why it has been impossible for others to replicate these results. If this error turns out to be an actual mistake Reinhart-Rogoff made, well, all I can hope is that future historians note that one of the core empirical points providing the intellectual foundation for the global move to austerity in the early 2010s was based on someone accidentally not updating a row formula in Excel.

So what do Herndon-Ash-Pollin conclude? They find "the average real GDP growth rate for countries carrying a public debt-to-GDP ratio of over 90 percent is actually 2.2 percent, not -0.1 percent as [Reinhart-Rogoff claim]." [UPDATE: To clarify, they find 2.2 percent if they include all the years, weigh by number of years, and avoid the Excel error.] Going further into the data, they are unable to find a breakpoint where growth falls quickly and significantly.

The Reinhart-Rogoff study has been a key ideological prop of pro-austerity polices. Check Reinhart-Rogoff's website for a list of the dupes economists who cited to it, who will presumably express their deep regrets and embarrassment, and promptly retract have a good laugh together next time they meet in the green room, exactly like the WMD guys did. Of course, austerity has led to the suffering of millions, and the deaths by suicide of many. But austerity was the goal. As along as the bottom line was right, nobody cared about the details. Never, therefore, have I been so happy to claim: "That's not a bug. It's a feature." This story is the WMD story of today's Neo-Depression. (Same post at Seeking Alpha.)

UPDATE Contact information:

Rogoff: Littauer Center 216
Tel: 617.495.4022
Fax: 617.495.7730

Phone: 609-258-4781
Fax: 609-258-5974 E

UPDATE Apparently, I went for the fun part. Oh well!

fwiw, the excel is the most fun error, but the post-WWII exclusions are the big problematic driver of their results:
Mike Konczal

(I went for the Excel error because Excel errors are so ubiquitous, as for example in Jippy Mo's London Whale trade.)

UPDATE Yglesias says that the original study was so tendentious that nobody will care about an error in it:

This is literally the most influential article cited in public and policy debates about the importance of debt stabilization, so naturally this is going to change everything. Or, rather, it will change nothing. As I've said many times, citations of the Reinhart/Rogoff result in a policy context obviously appealing to a fallacious form of causal inference. There is an overwhelming theoretical argument that slow real growth will lead to a high debt:GDP ratio and thus whether or not you can construct a dataset showing a correlation between the two tells us absolutely nothing about whether high debt loads lead to small growth. The correct causal inference doesn't rule out causation in the direction Reinhart and Rogoff believe in, but the kind of empirical study they've conducted couldn't possibly establish it. To give an example from another domain, you might genuinely wonder if short kids are more likely to end up malnourished because they're not good at fighting for food or something. A study where you conclude that short stature and malnourishment are correlated would give us zero information about this hypothesis, since everyone already knows that malnourishment leads to stunted growth. There might be causation in the other direction as well, but a correlation study woudn't tell you. The fact that Reinhart/Rogoff was widely cited despite its huge obvious theoretical problems leads me to confidently predict that the existence of equally huge, albeit more subtle, empirical problems won't change anything either. As of 2007 there was a widespread belief among elites in the United States and Europe that reductions in retirement benefits were desirable, and subsequent events regarding economic crisis and debt have simply been subsumed into that longstanding view.

Yep. I mean, it's not like we're talking scholarship here.


UPDATE FT Alphaville adopts a measured tone:

2) It’s very difficult to know whether the Reinhart & Rogoff finding actually changed the course of policy that was yet to be decided, or if it was simply used as a post-hoc justification for policy that was inevitable anyways. The paper certainly did provide intellectual justification for austerity advocates, but beware of exaggerated claims. It wasn’t the only paper to arrive at such a conclusion, and the R & R study had come under plenty of previous criticism.


Still, a case to consider — Olli Rehn’s citation of “serious economic research” (this paper) when defending “fiscal consolidation” in Europe this year, including the 90 per cent debt figure. Rehn’s stance was widely criticised in some detail at the time, of course. But we have to admit, it’s very easy for a politician to answer back with a single figure like this, whether or not the figure originally wagged the dog and inspired policy in the first place. And it’s problematic when public officials fail to appreciate the extent to which papers like these need vetting.


On "used as a post-hoc justification for policy," the paper clearly was. That doesn't absolve Reinhart or Rogoff of the scholar's ethical responsibility not to propagate error. On "wasn’t the only paper," normally I hate prostitution metaphors in political economy (they are unfair to prostitutes), but yes, in this case Reinhart and Rogoff weren't the only ladies of negotiable affection in the brothel that is neo-liberal economics. But they were and are what they are, despite the undeniable fact there are many others like them, aching to be them.

UPDATE Economist's View:

The work of Reinhart and Rogoff was a major reason for the push for austerity at a time when expansionary policy was called for, i.e. their work supported the bad idea that austerity during a recession can actually be stimulative. It isn't as the events in Europe have shown conclusively. To be fair, as I discussed here (in "Austerity Can Wait for Sunnier Days") after watching Reinhart give a talk on this topic at an INET conference, she didn't assert that contractionary policy was somehow expansionary (i.e. she did not claim the confidence fairy would more than offset the negative short-run effects of austerity). What she asserted is that pain now -- austerity -- can avoid even more pain down the road in the form of lower economic growth. Here's the problem. She is right that austerity causes pain in the short-run. But according to a review of her work with Rogoff discussed below, the lower growth from debt levels above 90 percent that austerity is supposed to avoid turns out, it appears, to be largely the result of errors in the research. In fact, there is no substantial growth penalty from high debt levels, and hence not much gain from short-run austerity.

UPDATE Dean Baker:

This is a big deal because politicians around the world have used this finding from R&R to justify austerity measures that have slowed growth and raised unemployment. In the United States many politicians have pointed to R&R's work as justification for deficit reduction even though the economy is far below full employment by any reasonable measure. In Europe, R&R's work and its derivatives have been used to justify austerity policies that have pushed the unemployment rate over 10 percent for the euro zone as a whole and above 20 percent in Greece and Spain. In other words, this is a mistake that has had enormous consequences. In fairness, there has been other research that makes similar claims, including more recent work by Reinhardt and Rogoff. But it was the initial R&R papers that created the framework for most of the subsequent policy debate. And [Herndon, Ash, and Pollin have] shown that the key finding that debt slows growth was driven overwhelmingly by the exclusion of 4 years of data from New Zealand.

UPDATE And here is ground zero: Herndon, Ash, and Pollin from UMass's Political Economy Research Institute:

Herndon, Ash and Pollin replicate Reinhart and Rogoff and find that coding errors, selective exclusion of available data, and unconventional weighting of summary statistics lead to serious errors that inaccurately represent the relationship between public debt and GDP growth among 20 advanced economies in the post-war period. They find that when properly calculated, the average real GDP growth rate for countries carrying a public-debt-to-GDP ratio of over 90 percent is actually 2.2 percent, not -0:1 percent as published in Reinhart and Rogoff. That is, contrary to RR, average GDP growth at public debt/GDP ratios over 90 percent is not dramatically different than when debt/GDP ratios are lower. The authors also show how the relationship between public debt and GDP growth varies significantly by time period and country. Overall, the evidence we review contradicts Reinhart and Rogoff's claim to have identified an important stylized fact, that public debt loads greater than 90 percent of GDP consistently reduce GDP growth.

Funny how scholars at "second rank" schools like UMass and UMKC are beating the shills and whores at Yale and Princeton like gongs.

UPDATE Krugman:

Holy Coding Error, Batman [T]he really guilty parties here are all the people who seized on a disputed research result, knowing nothing about the research, because it said what they wanted to hear.

UPDATE R-R respond. Shorter: Bad data and coding errors don't matter because others got the same results. Because scholarship! (Oh, and the first word in the response? "Ben." That would be Ben Smith of Politico, which shows you where this is going to be played out, for good or ill.

UPDATE Yglesias responds to R-R:

Carmen Reinhart and Vincent Rogoff have emailed a response to today's criticisms of their research on debt and growth and I've pasted the full text of their response below the fold. As I predicted, the criticisms change nothing. To sidestep this debate for a moment, I would note [Ugh. Beltway subjunctive] in particular that the entire debate completely fails to acknowledge the main flaw in the R&R research. As they themselves acknowledge in the response, the empirical correlation that we're arguing about is completely irrelevant to the policy debate at hand. They write that "we are very careful in all our papers to speak of 'association' and not 'causality'". This genuinely ought to settle the debate. Nothing Reinhart and Rogoff present, under any interpretation or any methodology offers any reason to believe that a high debt:GDP ratio causes slow growth. Yet much political rhetoric, including some from Reinhart and Rogoff themselves, presents their work as offering important policy guidance. It's great that when challenged they retreat to the more defensible claim that their work is actually irrelevant [Ouch!], but many policymakers and pundits seem to feel otherwise.

Ha ha! Say, where's Young Ezra on this? Waiting to weigh in and settle things?

UPDATE Which party can blame the other first?! From The Hill:

[R-R's] work has been cited not just by Republicans, but members of both parties training their fire on the nation's deficit. Former Senate Budget Committee Chairman Kent Conrad (D-N.D.) cited the study on the Senate floor in his farewell address before his retirement in 2012. Calling it a "landmark work," he used it to underscore his persistent push to shrink the nation's debt.

Now I need to find an Obama quote on debt-to-GDP ratio to go along with Conrad's.... Checking back in for another

UPDATE, Ars Technica points to the lack of peer review:

The paper's problems also highlight the value of peer review. Although the paper was included in the American Economic Review, it was included as a "Papers and Proceedings" article, taken from a conference and published without being reviewed. A peer reviewer might have sought the Excel spreadsheet at review time and could have exposed these flaws sooner. In turn, this may have prevented an apparently flawed paper from receiving the widespread attention that it got. The response is similarly unreviewed, so it too may be flawed.

Well, that's neo-feudalism for you! And actually, the response is reviewed: The spreadsheet is there for all to test. And it contains the errors cited. One does wonder whether the "editors" of the American Economic Review will issue a statement. "Mistakes were made," perhaps?

UPDATE Bloomberg lets the UMass authors put the "political" in "political economy," right where it belongs:

The findings by Reinhart and Rogoff “have served as an intellectual bulwark in support of austerity policies,” the University of Massachusetts academics said. The fact that their results “are wrong should therefore lead us to reassess the austerity agenda.”

Well, only if evidence is a concern. Let's be careful to caveat that.

UPDATE George Osborne, 2010 (via The Telegraph):

[OSBORNE] As Rogoff and Reinhart demonstrate convincingly, all financial crises ultimately have their origins in one thing – rapid and unsustainable increases in debt. As they write, "if there is one common theme… it is that excessive debt accumulation, whether it be by the government, banks, corporations, or consumers, often poses greater systemic risks that it seems during a boom.

And the Telegraph comments:

So is this merely an academic point, an obscure intellectual exercise with no political meaning? Not quite. If – and it remains an “if” – Rogoff and Reinhard are discredited, Mr Osborne has lost one of the intellectual props for his political-economic judgement. In essence, his position has become just a little more exposed. If the story of this government is the story of George Osborne’s era-defining economic gamble, then the stakes have just got even higher.

UPDATE Quartz on policy-maker buy-in. Not just Osborne!

Were other policymakers around the world paying attention? Absolutely: “[I]t is widely acknowledged, based on serious research, that when public debt levels rise about 90% they tend to have a negative economic dynamism, which translates into low growth for many years.” — European Commissioner Olli Rehn. “Economists who have studied sovereign debt tell us that letting total debt rise above 90 percent of GDP creates a drag on economic growth and intensifies the risk of a debt-fueled economic crisis.” — House Budget Committee Chairman and former Republican vice-presidential candidate Paul Ryan. “It’s an excellent study, although in some ways what you’ve summarized understates the risks.”— Former US Treasury Secretary Tim Geithner. [If Geithner, then Obama, but I'd want to be 100% certain Geithner's talking about the 2010 paper, not the 2009 book. A Google search doesn't turn up the original sourcing.] “[W]e would soon get to a situation in which a debt-to-GDP ratio would be 100%. As economists such as Reinhart and Rogoff have argued, that is the level at which the overall stock of debt becomes dangerous for the long-term growth of an economy. They would argue that that is why Japan has had such a bad time for such a long period. If deficits really solved long-term economic growth, Japan would not have been stranded in the situation in which it has been for such a long time.” Lord Lamont of Lerwick, former UK chancellor and sometime adviser to current chancellor George Osborne. “The debt hurts the economy already. The canonical work of Carmen Reinhart and Kenneth Rogoff and its successors carry a clear message: countries that have gross government debt in excess of 90% of Gross Domestic Product (GDP) are in the debt danger zone. Entering the zone means slower economic growth.”— Doug Holtz-Eakin, Chairman of the American Action Forum.

UPDATE Economix's Anne Lowry:

Flaws Are Cited in a Landmark Study on Debt and Growth Moreover, the argument floating around that the Reinhart-Rogoff research altered any country’s policy course – the idea that we have austerity because of a spreadsheet mistake – is dubious at best. When the global recession hit, many members of the policy elites in Europe and in the United States already believed that budget discipline begat growth. And countries that adopted austerity budgets often had to do so because of the vicissitudes of the market and the financing constraints of groups like the International Monetary Fund and the European Commission.

In short, no harm, no foul. The Very Serious People agree, and what's a little bad data and a few coding errors between friends?

UPDATE Daniel Akst at the WSJ is all "he said, she said:

Is 90% Really a Sovereign-Debt Red Line? A New Paper Says No Who’s right? You be the judge. Meanwhile, it’s unlikely we’ve heard the last of this.

Well, yes. Now you can be the judge, because the UMass people posted the data and the spreadsheet. What does that tell you?

UPDATE Reuters quotes the RR response:

"Of course much further research is needed as the data we developed and is being used in these studies is new," they said in a joint response by email. "Nevertheless, the weight of the evidence to date - including this latest comment - seems entirely consistent with our original interpretation of the data."

Well, copy edit problems here; "more" research isn't needed; research is needed. Bad data and bad coding aren't research (unless you confuse research with funding, of course). The question somebody should ask RR is this: Will all the data and the spreadsheets for their 2009 book now be made available? We need to know if the 2010 ("Growth in a Time of Debt") paper is an isolated incident -- sloppy work from a fall guy grad student, perhaps? -- or whether the methodological, er, issues with the 2010 paper are pervasive through their oeuvre. After all, fakers fake; they don't do it once and then stop.


Zeitgeist alert! First "The TA is to blame!" sighting:

UPDATE LA Times's Michael Hiltzik responds to RR's response:

In their response, R & R don't directly address [yep] the UMass critique that their math was erroneous. Instead, they argue that since the critique also finds lower economic growth in high-debt situations, the new findings are no big deal. The point, however, is that the corrected figures don't show anything like a break point in economic growth at the 90% ratio, which Rogoff and Reinhart cited.


R & R also point readers to a 2012 paper that they say validated their original findings. That paper, however, is more nebulous than they suggest. In it they found lower growth in high-debt-ratio conditions, but it also acknowledged that the high debt was associated with other growth-reducing conditions, including wars, banking crises and depressions. If that sounds like the developed world in 2008-2013, no kidding.

That underscores what has been one of the most frequent critiques of the original paper: Even if high debt/GDP ratios are associated with low growth, what's the evidence that the former causes the latter? [If RR have any, they'd better make the data and the spreadsheet public, eh?] It's been widely pointed out that it's just as likely, even more likely, that low growth drives higher government debt, rather than the other way around.

One reason is that lower growth will suppress GDP, thus directly affecting the mathematical ratio. Another is that low growth spurs governments to take on more debt as a recovery tool. Under both rationales, it makes sense to respond to economic slumps with more government spending, not with austerity. That's where fans of Rogoff and Reinhart, like Rep. Paul Ryan, have gotten things exactly wrong.

UPDATE Matt Bruenig at TAP dances on the grave:

For those in policy circles, it is hard to understate how big a deal this actually is. Pro-austerity forces obviously have separate motives for why they want to bring on the economic pain that has nothing to do with Reinhart and Rogoff. But for the last few years, this research has been providing the pro-austerity crowd intellectual cover for their policy preferences. Everywhere you turn in these debates, you are bombarded with Reinhart and Rogoff. While there were plenty of good reasons to dismiss their theory before now, the fact that their own data—when corrected for basic coding—cuts against them should officially render this theory dead. It is simply not the case that a government with a debt-to-GDP ratio over 90 percent is a death sentence for economic growth. Reinhart and Rogoff were wrong even on their own terms.

The theory will be dead when Obama kills it. Or rather, the theory will no longer be undead when Obama destroys its brain.

reinhart_rogoff_coding_error_0.png67.03 KB
Average: 5 (2 votes)


letsgetitdone's picture
Submitted by letsgetitdone on

Funny thing is. This latest criticism, misses a central point. That is, that Reinhart and Rogoff failed to distinguish nations sovereign in their currencies from nations that are not. So whatever association they came up with even without the "errors" would have been meaningless, since splitting these nations into these two groups would likely have produced entirely different associations or correlations. We would not hear this from Konczal or Yglesias, because that distinction isn't prominent in their own writing.

Btw, Lambert, nice biting piece. If you post it at NC, don't forget the fiat currency point.

Rainbow Girl's picture
Submitted by Rainbow Girl on

This is a flaming, not a smoking, gun. Thank you for the initial posting and the running updates.

"This will not be a bloodless process." (Barack Obama) No, it hasn't been.
No wonder there has been such false hysteria over looking for "WMDs" (in other continents) or "Al Quaeda" (or "Muslim") terrorists in America.

One big diversion to keep all eyes away from the REAL WMD.

I am extremely angry.

Martin Finnucane's picture
Submitted by Martin Finnucane on

I am extremely angry.

Feeling ya.

Rainbow Girl's picture
Submitted by Rainbow Girl on

"The fact that their results 'are wrong should therefore lead us to reassess the austerity agenda' [...] and immediately consider aggressive measures to compensate all individuals whose true lives have been DIRECTLY damaged by Rogoff and Reinhart."

Time for reparations, Obama. Do you hear us?

In addition, now that the fake Rogoff-Reihnart cat is out of the bag, it's time for Mr. Obama and all members of Congress to put down their Grand Bargain poison-pencils. The jig is up.

(1) Singled Payer (Medicare for All) with all savings passed through to patients.
(2) Triple all payments made through New Deal and post-New Deal safety net programs -- that includes Social Security -- and establish COLA at the rate of real inflation in the real economy.
(3) Reverse all measures currently in effect to destroy USPS and add a USPS Bank with savings accounts paying 5% interest and loan rates at 1% or less.

There's much much more. These three items (in addition to cash reparations noted above), however, are emergencies and must be attended to IMMEDIATELY.

Mr. Obama, your failure to invoke your emergency powers to implement the above measures by Presidential Decree will have to result in your impeachment and immediate removal from office.

Martin Finnucane's picture
Submitted by Martin Finnucane on

This is the academic version of unstocked shelves and long lines at Wal Mart (devolution): crumminess all around us, everything going to seed, even our enemies. We weren't beaten by a Smith or Ricardo, or even a Friedman or Hayek. These bozos, with their crummy spreadsheet, sufficed.

Martin Finnucane's picture
Submitted by Martin Finnucane on

On the "TA is to blame" sighting: the same guy in the same conversation also reminded us to "never assume malice where incompetence will suffice." I hate when people say that. Yes, why assume that the arsonist meant to burn down the house, when we can explain the fire as an instance of pyromanic somnambulism? Try that in front of a jury, pal. But of course, the important white guys in ties don't end up in the dock, in front of the jury. Rather, we "never assume malice," even when we clearly see them pouring gasoline on our front porch, match in hand.

Another variant on this theme is the "tin-foil hat" argument: we shouldn't, or can't, point the finger of blame at the important white guy in a tie - to do so would make us sound like tin-foil hat wearing crazies. Hate that one too. The world is crazy, if hacks like R&R are respected economists, but it is "loony tunes" to suggest that something is rotten when the looters pushing for austerity are the same class that both created and benefited from the GFC. Get me off this world, or at least give me a tin foil hat.

letsgetitdone's picture
Submitted by letsgetitdone on


After expressing outrage, highlighting the new criticism, and also recalling a previous paper by Yeva Nersisyan and himself. He says:

Here’s the bigger problem highlighted by Yeva and Me: they do not know what they are talking about. Sovereign countries that issue their own floating currency cannot be forced into involuntary default no matter what the debt ratio. For that reason, even if their results had not been tainted by bad research, it would have been irrelevant to the situation of any country that issues its own floating currency, such as the USA, the UK, Japan, and so on. No matter what the debt ratio, a sovereign government that issues its own currency can choose to grow the economy.

That was the lesson that should have been learned. Here’s how we ended our critique:

When it comes to a sovereign government’s budget deficit and debt there are no magic numbers or ratios that are relevant for all countries and all times. There are no thresholds that once crossed will be unsustainable or lead to lower growth. The government’s budget balance in most advanced nations is highly endogenous and is merely the other side of the coin of the nongovernment sector’s balance. The public deficit is the result of the private sector’s willingness to net save and net import.

Modern monetary theory is often interpreted as claiming that there is no real limit to the government’s ability to spend or that the government should run up deficits. Of course there is a limit to the government’s ability to spend and of course it shouldn’t spend an infinite amount. Yet, the sovereign government is not constrained financially, which means that it can never face a solvency issue. Still, it is certainly constrained in real terms meaning it can face another kind of sustainability issue: how much of the nation’s resources ought to be mobilized by government? Given the level of resources that the nongovernment sector wants to mobilize, how large should the government’s deficit be to mobilize the rest?

More than five decades ago, Abba Lerner gave the answer to this question. If there are involuntarily unemployed (we would add underemployed) people it means the deficit is too low. The government should either cut taxes or increase spending. It is certainly debatable which one is a better policy, but that’s beyond the scope of this paper. When is the deficit too large? When it’s over 3%, 7%, 10%? Again, there is no magic number and anyone who comes up with a universal number simply misunderstands the modern monetary regime and macroeconomics. In opposition to magic, Lerner proposed “functional finance”—the notion that the federal government’s budgetary outcome is of no consequence by itself, but rather, what is important is the economic effects of government spending and taxing. When total spending in the economy, including government spending, is more than what the economy is able to produce when employed at full capacity, the government should either lower its spending or raise taxes. A failure to do so will lead to inflation. So inflation is the true limit to government spending not lack of financing. Government debt is merely the result of government deficit and hence the same applies to debt as well.

Lesson to learn: ignore the Ivory Tower economists who recommend austerity and warn of “unsustainable” budget deficits in the case of sovereign currency nations. They know not of what they speak.

affinis's picture
Submitted by affinis on

“Funny how scholars at "second rank" schools like UMass and UMKC are beating the shills and whores at Yale and Princeton like gongs.”

Having spent my adult life in academia, this is not surprising to me at all. Though there’s plenty of great work that comes out of the elite private schools, there’s also a darker side. Places like Harvard, Yale, Princeton, Stanford, etc. tend to hire and retain fame-seeking academics who are good at promoting themselves. There’s on average (need to emphasize that this is obviously not true of all) an increased level of narcissism among the faculty (good at charismatic leadership and “vision” but not too attentive to details) and pressure to pump out lots of publications. Both these factors lead to lots of sloppy errors of exactly this kind. And there’s more tendency to manipulate data to generate “exciting” findings. In a way, I think this same set of problems is reflected in the very high error and retraction rates for papers in the highest profile scientific journals (such as Science, Nature, and Cell).

Submitted by Hugh on

When I write up my monthly analyses of the jobs report, I am dealing with a lot of numbers on the fly and trying to get both the numbers and what they mean right. I have this horror of making mistakes in the numbers. Early on I had a conceptual error in the disemployment figures which when I found it, I announced and corrected. Since then, there have been some typos but, knock on wood, the numbers I have reported have been correct. I have not had to use Excel for this, but I have used it for various other investigations and research.

I say all this because having worked with this kind of data, it is really hard for me to see how Reinhart and Rogoff could have made this kind of error and, having made it, let it persist for so long. First off, they weren't under any time constraints. Second, I usually check the data ranges against what is in the formulas more than once (the RR error). Or I repeat the whole process and compare the results. I usually test the data from a variety of angles. So an error in one test will show up as a discrepancy in another, meaning I go back through the calculations until I find what happened. I compare the results of my calculations to past findings. I eyeball results. If they don't seem right. I redo them just to be sure. And over time new data may come in. Incorporating it would entail a recalculation, and again this would expose the initial error.

The point I am trying to get across here is that this was not a single error. It is an error that had to pass through, or should have had to pass through, numerous filters, one or more of which should have caught it, and this happened with not one but two supposedly recognized authorities working with the data. I don't know about you, but if someone gave me an analysis, especially one I was going to put my name on, I would see if I could replicate it, if not in whole, at least at all the critical junctures.

So while I admit that such an error is possible, it remains incomprehensible to me that it occurred.

Beyond this, I have problems with one-size-fits-all analyses, whether in space or time. Not all countries are the same, nor are their economies and monetary systems directly comparable. Similarly, comparing recessions over the last few centuries is a sterile exercise because the underlying politics and economies are so different.

Finally, Reinhart and Rogoff should have insisted from the beginning that their debt to GDP ratio threshold (if it had actually existed) was indeed a correlation, not causative. And they could have done this precisely by stating it the other way around, that countries with slow growth often also have higher debt. They could then have gone on and investigated why that is the case (bubble collapses, governmental support of the banking sector, support of the social safety nets, etc.). Instead they stated it in a way that invited an austerian interpretation and never made any real effort to challenge it.

Jay's picture
Submitted by Jay on

I'd like to suggest that:

  • R&R should henceforth be referred to as "the discredited Reinhart and Rogoff paper."
  • That Reinhart and Rogoff should be known as "the arithmetically-challenged Reinhart and Rogoff."
  • Justifications for austerian policies rely on "Enron-style accounting."

affinis's picture
Submitted by affinis on

Andrew Gelman makes a valid point about Reinhart and Rogoff's defensive responses: "unless they want to enter the competition for the lamest, grudgingest, non-retraction retraction ever, I recommend they start by admitting their error and then going on from there."

Though my work is in a very different field than Hugh's (I'm a geneticist/evolutionary biologist), the procedures I follow with data and spreadsheets are very similar to what Hugh describes for himself. Those error checks/sanity checks do take a bit of time, but they're essential if you care at all about truth and publishing something that's valid. The only conclusion I can draw from a case of extreme sloppiness such as this is that R&R didn't really care too much about getting it right/truth - perhaps driven by preconceived narrative, details of the data be damned.

letsgetitdone's picture
Submitted by letsgetitdone on

with empirical data and statistical research myself, and I agree entirely with comments made by Hugh and affinis (in two comments). To amplify some things affinis said a bit more, I think there's more than sloppiness, cavalier treatment of data, TAs, failure to make checks that should have been built into their procedures or other issues of that sort operating here. The "axe to grind" bias of the research is very transparent now that the data are public. They made methodological decisions designed to get the result they wanted. Once they got it, they apparently had no interest in challenging it to see if it would hold up under various tests. It was a surprising result bound to be questioned by others who found the conclusion uncongenial to their views. Care for their own reputations, and for the people their result was likely to affect, as well as for the demands of objectivity should have gotten them to perform their own challenges to their findings BEFORE they published.

Of course, they did no challenging. When they began to get criticisms from MMT quarters pointing out that you cannot lump together fiat currency sovereigns and nations on gold or other commodity standards, nations using other people's currencies, nations pegging their currencies to others not their own, and nations owing debts in currencies not their own, they still persisted in their claims for the validity of their findings and refused to go and look at their data again to do the simple dummy variable analyses that would have immediately shown whether their conclusions held up when they took these factors into account. And all the while they held their data close so no one could second guess based on their own data. All this is bias. It is not science. It is more like a case made by a lawyer who wants to prove a point, then it is like a scientist or any other investigator who is trying to find out the truth.

I think they were after power, prestige, money, and status, and not the truth. They are corrupt, and their corruption is what this incident is about.

Miguel Sanchez's picture
Submitted by Miguel Sanchez on

I don't think they were the least bit concerned about whether the data was accurate or the result of a spreadsheet error because the conclusion just felt so gosh darn right! Any errors that might come to light later wouldn't matter because they would be trivial, you see, and couldn't possibly alter the entire conclusion of the research, which had to be true because it just felt so gosh darn right.

It reminds me very much of the claims that there were WMDs in Iraq. The Bush admin didn't care that the data supporting that claim was weak because they just knew that Saddam, being the bad awful guy that he was, would have the WMDs, and that once his trove was found, as it inevitably would be, Bush would be vindicated and the weakness of the original claims forgotten.

It's all part of the mindset that starts with a preconceived notion of what is true, then works backward from there seeking evidence only to bolster such claims for the sake of the foolish skeptical rubes who think evidence matters. This mindset, which is a hallmark of modern conservatism, has no real understanding of the concept that evidence is the only pathway to truth. Right thinking people already know what is true and what is false and don't waste time worrying about the mere formalities of 'evidence' or 'data'. Yes, it's definitely not science.

Submitted by lambert on

A fine example of the "more in sorrow than in anger" approach:

I was going to post something sort of kind of defending Reinhart-Rogoff in the wake of the new revelations — not their results, which I never believed, nor their failure to carefully test their results for robustness, but rather their motives. But their response to the new critique is really, really bad. ...

[here beginneth and endeth the takedown]

So this is really disappointing; they’re basically evading the critique. And that’s a terrible thing when so much is at stake.

letsgetitdone's picture
Submitted by letsgetitdone on

NC wouldn't accept this comment when submitted. So, I'm submitting it here, and posting a short reply at NC to drive traffic to it. It's on the same subject.

Does MMT make any data point choices in its framework— or put in a slightly different way—does MMT ever make any choices as to where to conceptually cut their focus?

You mean case choices? As the Sector Financial Balances (SFB) model applies to some economies but not to others? No, as far as I know, it doesn't.

For example, would you admit to a conceptual choice as to where to cut your historical focus in your understanding of the history of money or would you admit that a conceptual choice is at work in your emphasis on state fiat money creation rather than private bank-created money?

Yes, I would in this area. MMT is obviously about economies that use money. As far as admitting that MMT's emphasis is on state fiat money creation rather than on private bank-created money, I don't believe that's true in general. In the MMT framework the banking system includes the central bank and is closely related to Treasury taxing and spending and Congressional appropriations. It's all one system and MMT encompasses all of it.

Are such choices in any sense political?

All theories involve value judgments. Neither science nor ordinary discourse is value free. See here.

The value judgments made may well relate to politics. MMT's selection of interesting problems, interesting concepts and variables, and evaluations about whether evidence is strong enough to refute certain economic theories all involve values and therefore can all have political implications. The more significant the problems for human beings the more likely they are to be "political."

Is it conceivable that your perspective on the history of monetary operations is as circular as the apparent perspective of R and R because of the type of choices( for them–excluding certain time periods and non-standard weightings) they have apparently made?

Sure it's conceivable, but what do you mean by circular? If you're making the old point that MMT is no more justifiable than the neoliberalism of RR, then I certainly agree. But from my point of view no theory can be justified, nor is there any need for its justification.

All a theory, or a paradigm, or a conceptual framework must do is to be open to and survive any and all criticisms thrown against it better that does its competition. If it does that then it's a better theory, or framework, or approach or whatever.

In my view MMT is currently surviving criticisms against it a lot better than the dominant school of neoliberalism, or the Austrian nonsense, or any of its current competitors. So, that's why I'm interested in telling others about it.

Do all of us(of whatever political persuasion) tend to conceptual slicings that are supportive of our political sympathies?

No, assuming that you're not defining political persuasion to mean everything in the world. Some of us are committed to finding truth in the old sense of correspondence with reality and we make every effort to do our conceptual slicings to support this quest. This doesn't mean our political persuasions don't influence our choice of problems, selection of variables and assessment of evidence. They may well do that. But that doesn't mean they GOVERN our conceptual slicings.

Could it be that both of these perspectives as well as my own and as well as yours are looking for adequate premises from which to infer conclusions already and independently accepted because of one’s sympathies?

The issue isn't whether we're looking for premises to infer the conclusions we want to come to as much as it is whether we're open to criticism of such conclusions and premises and whether we're open to alternative theories and points of view. R and R were not as evidenced by their keeping their data secret for so long, beyond the sphere of criticism. R and R wouldn't check their own work. They wouldn't challenge it. Instead they ran and hid from criticism, until they could do so no longer.

Could it be that because there our always choices involved as to where to situate a problem to begin with that all of us tend to articulate positions that do not exist independently of our formulations?

Of course, our articulations are not independent of our formulations; but nor are they determined by them, because we can always invent new formulations and articulations. You're stating "the myth of the framework." A myth effectively critiqued some years ago now by Karl Popper.