For Government That Works, Call In the Auditors
Environmental regulators in Gujarat, one of India’s fastest-growing industrial states, found themselves in an implausible situation a few years ago: Every single city breached national air quality standards. And yet environmental audits kept finding that factories met pollution limits.
So the Gujaratis hired some researchers from Harvard and the Massachusetts Institute of Technology to carry out an experiment, changing the way the audits were made. Instead of hiring their own auditors, companies had auditors assigned to them randomly. Instead of being paid by the companies they audited, auditors drew a fixed fee from a pool that all companies paid into.
Measured compliance rates abruptly plummeted. But once the new system was in place, the real emissions from polluting factories finally started to decline. The Gujaratis kept the new approach.
Continue reading the main story
Q. and A.: Lab Explains Its Evaluation of WeatherizationOCT. 6, 2015
Economists Critique Report on Weatherization Assistance
“When fact-checking is not done in an independent way, there is a long history of things turning out the way the entity being fact checked wants them to turn out,” said Michael Greenstone of the University of Chicago, a former chief economist for President Obama’s Council of Economic Advisers who was one of the researchers involved in the study. “Until you change the incentives, this will not change.”
The problem may seem remote, but it turns out that the same incentives apply in the United States, even in programs that, at first glance, appear to provide an unmitigated benefit.
Last month, the Energy Department released an extensive report assessing the impact of the federal weatherization program, which was begun in 1976
to shield the homes of low-income Americans from the elements, save them money on heating bills and improve energy efficiency.
It concluded that weatherization — insulating homes, changing boilers, plugging leaky windows and the like — was a stellar investment. Not only were the energy savings substantially larger than the cost of weatherizing homes, the report found, but the gains soared even more once the broader impacts on health were taken into account.
“The results demonstrate that weatherization provides cost-effective energy savings and health and safety benefits to American families,” the Energy Department announced.
But do they? When Professor Greenstone and two other independent economists looked under the hood — not a trivial challenge, given the report’s 4,500 pages — they found a collection of idiosyncratic choices and unorthodox assumptions that severely undermined the credibility of the enterprise.
A column by Eduardo Porter that explores the world’s most urgent economic challenges.
Getting to $100 Billion in Climate Change Aid
Education Gap Between Rich and Poor Is Growing Wider
A Migration Juggernaut Is Headed for Europe
Corporate Efforts to Address Social Problems Have Limits
Donald Trump’s Shaky Grasp on Immigration
See More »
In the end, they concluded, the government research effort, which was led by the Energy Department’s own Oak Ridge National Laboratory, cannot tell us whether weatherization is a fabulous program or a waste of taxpayer dollars.
Consider the energy savings. According to Oak Ridge’s analysis of the program for the year 2008, over time they amounted to 1.4 times the program’s cost.
The cost-benefit analysis does not hold up well under scrutiny, though. “Costs” exclude all sorts of things, including administrative and training expenses, according to Bruce Tonn, who led the research at Oak Ridge.
Continue reading the main story
Moreover, the calculations did not allow for wear and tear, assuming that efficiency improvements hum along flawlessly for 20 years. And they relied on a generous calculation of the present value of future savings that assumes money loses value at a much slower pace than would be implied by the interest rate that determines the cost ordinary Americans face to borrow money now to pay in the future.
The estimates of nonenergy benefits — $14,148 per weatherized home in health, safety, productivity gains and the like — were even more problematic.
Unlike the energy savings, these benefits were not, in fact, measured. For instance, to assess the gains from lower mortality — almost $5,000 a weatherized home — researchers did not investigate how many deaths were prevented by weatherizing homes. Productivity gains from better sleep — another $3,142 in net present value per household — were not measured, either.
Instead, the researchers asked a sample of people questions like this: “In the past 12 months, has anyone in the household needed medical attention because your home was too cold (hot)?” The answers were projected onto national estimates of how many calls for medical attention lead to hospitalization, and then onto national estimates about the share of hospitalized people who die.
Oak Ridge’s report acknowledges that some of the estimates are more speculative than others, ranking them in three tiers of plausibility. Its bottom-line finding that the benefits amounted to more than four times the cost of the program included only $7,823 in health benefits that it considered the most viable.
The analysis might have gone unnoticed. But the Energy Department offered it as refutation of an earlier study written by Professor Greenstone, along with Meredith Fowlie and Catherine Wolfram of the University of California, Berkeley, who concluded that the energy savings produced by weatherization did not justify the costs.
The authors, determined to defend their own work, delved into the government analysis. They hired a graduate student to pore through the data. “Many of the D.O.E.’s conclusions,” they wrote, “are based on dubious assumptions, invalid extrapolations, the invention of a new formula to measure benefits that does not produce meaningful results, and no effort to evaluate statistical significance.”
To assess the impact of a government program on health, researchers usually take a sample of people and measure their health before and after the program. To ensure gains in this “treatment” group were caused by the program and not something else, they are compared to changes in a “control” sample of people who did not benefit from the program.
But the researchers at Oak Ridge did not assemble a “control” group of nonweatherized homes. So they had to make do without the usual checks. They surveyed the “treatment” group before and after their homes were weatherized and measured the difference in their answers to questions about their health and well-being. Then they measured the difference between the answers of the group of families in preweatherized homes and those of another set of families whose homes had been weatherized the year before. Finally, they averaged these differences.
Continue reading the main story
Continue reading the main story
“To the best of our knowledge, this approach has never been used in any textbook or research paper previously,” noted Professors Fowlie, Greenstone and Wolfram in a blog post critiquing the Energy Department’s report. “The approach to estimating nonenergy benefits is unrecognizable, and we believe the resulting estimates have no meaningful interpretation.”
What makes the exercise seem particularly odd is that the researchers ignored things they did measure. The study concluded there were big health gains from reduced thermal stress, but its field study of indoor air quality found no meaningful change in the temperature of weatherized homes.
The field study also found no significant changes in carbon monoxide. And it detected an increase in radon and formaldehyde levels. Yet the overall cost-benefit assessment reported benefits from decreased carbon monoxide poisoning and omitted the potential impact of higher concentrations of radon or formaldehyde.
“The report does not address longer-term nonenergy impacts,” Dr. Tonn explained. “It may take many years before exposure to radon could result in a health impact.” Still, the omission of effects pointing in the “wrong” direction cannot but invite skepticism.
Assessing the impact of weatherization is important. It is the federal government’s largest effort to improve residential energy efficiency, a critical component of its strategy to combat climate change. What’s more, given Congress’s relentless effort to squeeze the federal budget, it’s important that the government does not waste money and gets the most bang for our bucks.
In 2011, President Obama issued an executive order that called for regulatory agencies to carry out retrospective evaluations of the impact of regulations. Last month, the president’s order to include the insights of behavioral science in policy design again underscored the need to “rigorously test and evaluate” the impact of these insights to discard the bad ideas and keep the good.
Yet tough evaluations are tough to come by. A government agency that stands to lose a chunk of its budget if it finds a weakness in its programs is less likely to look too hard. An auditor commissioned by that agency to unearth the flaws in its programs may not dig as forcefully as he should.
This is not just a government shortcoming. Consider the many Triple-A ratings Moody’s and Standard & Poor’s produced on mortgage-backed securities at the behest of the firms that issued the securities.
“Self-evaluation is very difficult and runs the risk of appearing motivated by self-interest, rather than true assessment,” Professor Greenstone told me. “Evidence-based policy-making requires independent evaluations, and then acting upon the results, whatever they may be.”
This may seem like a rarefied dispute, but it’s worth getting right. Why? Ask the Gujaratis.