Archive | February, 2013

Straw Poll Fallacy

Good think tanks do research, and they also do advocacy, but think tanks that fail to make any distinction between the two squander valuable reputational capital.

Last Friday, my former MI colleague, Josh Barro, scolded the Florida-based James Madison Institute for conducting a “push poll” about the state’s federally-subsidized Medicaid expansion plans.  “This isn’t a poll designed to figure out how Floridians feel about the Medicaid expansion,” Barro complained, “it’s one designed to get them to say they oppose it, so the organization commissioning the poll can say it’s unpopular.”

Cato Institute health policy guru Michael Cannon, also a former colleague of mine, had apparently reviewed the poll questions for the James Madison Institute before the poll hit the field.  Cannon fired back:

Medicaid expansion is not a benefits-only proposition. When a poll only asks voters about benefits, the results are meaningless. Yet to my knowledge, JMI’s poll is so far the only poll that has asked voters about both costs and benefits. All other polls—for example, the hospital-industry poll Barro cites—ask only about benefits, as if the costs don’t exist or shouldn’t influence voters’ evaluation of the expansion. Those polls are “push” polls, while JMI’s poll is the only honest poll in the field.

I consulted an experienced GOP-leaning political pollster in the Washington, DC area to get the skinny.  The pollster, responding on condition of anonymity, expressed “serious concerns about the poll.” To wit:

First, it’s not a true survey of registered voters, because they focus mostly on pulling from registration lists those who voted in at least two of the last four elections. You can’t say that’s representative of Florida registered voters, though you could say its representative of likely voters. That’s a distinction that should be made clear, as it will bake in a slight right-leaning skew compared to straight-up registered voters.

I stopped reading and started writing this email when I hit that first debt question. A good poll would have asked a more “clean read” without loading up a big message before the ask about how important the debt is. The interviewer says “well everyone else cares about the debt, so, how concerned are you?” Really not good. This is the kind of question you push further down in the questionnaire as a message test, not as a legitimate gauge of concern about debt.

Then I got to the question [posed as] “some say we need reform” vs. “some say we need to preserve a government program.” How [often do] Democrats actually say, “we must preserve a government program!” Never. They say, “we must preserve needed health services for our poorest citizens,” etc. A good poll puts our best message against their best message. Already, the poll is putting up a weak version of the opposition’s position.

The point thus goes to Barro, though I’m sympathetic to Cannon, who is not a pollster and was only asked to review these questions for the accuracy of their substantive claims about Medicaid.

Such bad methods reflect poorly on the James Madison Institute, which holds itself out to be a research and educational organization, complete with a Research Advisory Council primarily composed of university-based social scientists. Think tank research isn’t expected to be peer-reviewed academic journal fodder, but it usually aspires to inform the public policy debate by telling us something new about the world we live in.  Think tank findings are often presented in light of researchers’ prior ideological commitments, but they should not merely be talking points in support of predetermined conclusions.

Not surprisingly, a James Madison Institute press release reveals that a division of the Florida-based public relations* firm Cherry Communications conducted the Medicaid expansion poll under contract.  “[While] a polling firm’s first goal is to create situational awareness,” the DC-area pollster explained, “a PR firm’s first goal is to create good headlines. These are each valuable but are not the same thing.”  Nor are they necessarily mutually exclusive:

There are really two different ways to approach designing a poll. One is if you want an accurate read on public opinion to guide strategic decision making. The other is to “message test” and to figure out how best to move opinion and build a communications plan. You can do both in one survey as long the “clean read” part comes first.

The fundamental problem here is that this poll was conducted with public release in mind and to show right off the bat that conservative messages on the issue work. This is a PR firm’s goal clearly. There’s no time taken to get the clean read.

The James Madison Institute hasn’t yet responded to my request for comment, but it isn’t hard to surmise what happened here: the communications department probably commissioned a poll as a way to get airtime for the Institute’s message on Medicaid expansion.  But a poll isn’t just a message.  A poll is a social scientific method, which why a lousy poll from a think tank casts doubt on the quality of its other research.

Journalists and policymakers afford more weight to think tank research than they do to press releases from PR firms because think tanks aren’t supposed to just spin.  The James Madison Institute may have rationalized this survey as the digital equivalent of liquid courage for skittish pols, but it should worry instead about what techniques like these suggest about its institutional values.  Reputation matters, because media and government consumers often don’t have the time or expertise to independently assess the quality of every report.  I am less likely now than I would have been last week to take anything in the James Madison Institute’s new policy brief on Medicaid expansion at face value, because I have reason to question the organization’s commitment to good research methods.

UPDATE:  I have just been informed that the Cherry Communications website I linked above belongs to a different firm with the same name as the “Cherry Communications” referenced in the James Madison Institute press release, whose division, “Public Insight,” conducted the Medicaid expansion poll.  I have eliminated the incorrect link, and I apologize to both firms for the error.

The James Madison Institute has offered some comments regarding the poll, and Jim Cherry of the Cherry Communications whose division conducted the Medicaid expansion poll has offered to comment also, so stay tuned for a follow-up post.

*Cherry Communications is really better characterized as a Republican political consulting firm specializing in phone-based services such as voter identification calls, persuasion/advocacy calls, get-out-the-vote calls, surveys, and polls.  It’s website is here.

About This Blog

A recent leadership struggle at the Cato Institute and a changing-of-the-guard at the Heritage Foundation have sparked a lively public debate about what think tanks do, how they do it, and about whether or not it is possible to do what think tanks do in an ethical manner. This blog will explore that last question from an optimistic perspective:  that it is possible for think tanks to maintain high ethical standards, and that most think tank executives and scholars on all parts of the political spectrum strive conscientiously to do so.  I am less certain that those of us in the public policy world know as much as we would like to know about how to accomplish this goal.

Different think tanks have vastly different institutional missions, cultures, and funding arrangements, but all appear to share a central challenge: reconciling scholarly aspirations with the imperative of relevance.

On one hand, think tank research usually aspires to be scholarly in the sense that it makes a new contribution to the public policy debate.  Wonks need not use methods associated with traditional academic disciplines, and indeed academic methods are often unsuitable for answering broad, practical questions about what to do.   Nonetheless, policy analysis is a highly-skilled job.  Tasks such as modelling a particular legal, social, or economic process, gathering and presenting new information about the prevalence of some social problem, or predicting the likely effect of a proposed reform can be done well or poorly by a set of fairly objective standards, and doing them well is the only way to come up with useful new insights.

Principled political commitments can and do inform empirical inquiry.  They are often the basis for deeming a research question important, and they are relevant to a scholar’s ultimate recommendations.  But they can’t replace empirical inquiry.  For this reason, I believe that insofar as think tanks hold themselves out to be engaged in the pursuit of new policy knowledge, they have an institutional obligation to establish working conditions for scholars that foster and reward good epistemic practices.

Think tank executives work very hard to accomplish this task, but it is an inherently challenging one, because think tanks are not supposed to produce brilliant analyses that no one reads.  They are on a mission to change the world we live in by producing research with “impact.”  Think tank scholars are thus expected to publish work that gets attention from the media and serious consideration from policymakers.  If there is no constituency within these two groups that is receptive to a scholar’s findings, those findings may receive little notice, leaving that scholar’s career languishing.

It can also be professionally dangerous for think tank scholars to generate findings that attract the wrong kind of notice, or which are are deemed helpful to the wrong people.  Some think tanks are more explicitly committed than others to advancing a particular notion of a just society through their work.  But insofar as all think tanks are committed to doing work that has an impact on public policy, it’s implausible that even the most ecumenical “university without students” is utterly institutionally indifferent to the kind of impact its scholars have.  Because public policies affect people’s lives, such indifference might even be considered irresponsible.  A think tank has a legitimate interest in choosing to support research that is likely to advance its institutional conception of positive social change.  Moreover, insofar as it solicits donations on this basis, it has an obligation to its supporters to do so.

These institutional imperatives, however reasonable, create a moral hazard for think tank scholars.  To be ethical, a think tank scholar must adhere to good epistemic practices regardless of where they lead her.  To be successful, she must generate findings that will interest constituencies in media and government and arrive at conclusions that do not appear to undermine her employer’s overarching institutional commitment to advance the cause of a just and humane society.  In my experience, scholars work very hard to maintain good research practices amid potentially conflicting professional imperatives.  But it is not always an easy task, and widespread reluctance to discuss this challenge can give rise to a sense of isolation that makes it more difficult.

I hope that this blog kindles a constructive conversation within the public policy community about how best to meet the ethical challenges of our work, because think tanks are valuable institutions.  While academics often focus on methodological minutiae, think tank scholars address pressing public questions.  While academics sometimes conceal the ideological convictions that inform their research, think tank scholars freely disclose their prior commitments.  While academics frequently write only for each other, think tank scholars magnify the impact of important, university-based research by translating it for a more general audience.  I’ve spent most of my career as a think tank scholar because I believe that think tanks have an important positive impact on the policy-making process that academia cannot replicate or replace.

I decided to start writing about think tank ethics because I believe that a community of smart, high-minded people who have a complicated ethical challenge in common can figure out how to meet that challenge even better if we talk to each other about it.  This blog will offer news analysis of events related to think tank ethics and governance, highlights from interviews with some of the many leading think tank executives and scholars who have generously shared their time and insight with me, and some preliminary analyses and proposals about which I’d like to receive feedback before they become part of the book that I will soon begin to write on this subject.  Please keep me on my toes with critical engagement in the comments or, better yet, at your own personal or institutional blog.  A competition of ideas on the subject of think tank ethics and governance can only benefit the institutions on which so many of us in think tanks, academia, government, media, or philanthropy rely for professional fulfillment, research, insight, or positive social change.