Public policy research is often methodologically mediocre. You might think, therefore, that more methodological criticism would help! Unfortunately, I don’t think it will be that simple. Technical critique is too often an emergency escape hatch for tired ideologues reluctant to litigate serious challenges to their principled commitments. Paradoxically, research quality may suffer as a result. A drained swamp lowers all boats.
Nearly every activity, including research, can be evaluated according to two types of standards: internal and external. Internal standards define the activity engaged in, while external standards evaluate that activity in light of the purpose it serves, the values it reflects, or its likely practical effects. For simple activities, internal standards are less controversial than external standards. We can usually more-or-less agree about what some activity involves in order to determine whether someone is or isn’t engaged in it. External standards are often controversial because they reflect disputed moral, political, or aesthetic values.
“Speech,” for example, is the activity of verbally expressing ideas using language. This is an internal standard for speech, and there are various ways in which our efforts to speak may fall short of meeting it. An infant who cries for a bottle expresses herself verbally, but because she does not know how to articulate words, her cries fail to qualify as “speech.” If you have laryngitis, you may form words, but if no sounds result, you, too, are failing to speak. If my mouth is numb following a dental procedure, I may emit sounds but have difficulty forming words. You might say that I am “speaking badly” if my mumbling efforts are partially understandable. In the context of an internal standard, “bad” speech lies on a continuum between what is inarguably speech and what fails to be speech at all.
But there is another, completely different way in which speech can be “bad.” Speech is often called “bad” if it is spiteful or inaccurate. Speech can also be bad in the sense that it promotes harmful behaviors or unjust policies. These are all external standards for speech. To raise the question of whether or not some particular speech is bad by some external standard is to raise philosophical questions about the validity and importance of that external standard as well as empirical questions about how the speech in question measures up. Speech can be very, very bad by an external standard while being good by the internal standards of speech. It makes perfect sense to say that Mussolini gave “bad speeches,” while at the same time conceding that he “spoke very well.”
Research of all types can be evaluated by both internal and external standards. Ideological commitments are external standards for the wonkish activity of public policy research. They should therefore function as figurative book-ends: as a basis for evaluating the worthiness of a particular research question, and as a basis for recommendations in light of a research result. In between, a wonk must adhere to the internal methodological standards applicable to public policy research in order to do her job well.
Because research is a complicated activity, its internal standards are disputed among experts and poorly understood by the public. Most public policy research combines ideologically-charged subject matter with contestable methodological choices. The result is a grave moral hazard: critics with fundamentally ideological complaints can till once more the exhausted soil of first principles…or, they can apply their efforts to the more pliant field of methodological critique!
Denying the validity of inconvenient research frees advocates from the challenging task of accommodating inconvenient facts. As Economist blogger Will Wilkinson explains, this rhetorical strategy is the chattering class’ current Nash equilibrium:
Perhaps it’s wishful on my part to think, as I do, that most economically literate observers really do understand that raising the minimum wage will screw up the prospects of a fair number of poor young workers. Those who favour raising the minimum wage anyway just think that, all things considered, that’s a price we ought to be willing to pay. But they can’t say that, just as second-amendment enthusiasts can’t say that an occasional grim harvest of kindergartners is a price we ought to be willing to pay for the freedom to own guns.
Charges of ineptitude have become so pervasive that the research signal gets lost in the media noise. Every study bearing on a controversial issue, if it is reported on at all, will be criticized by ideological hostiles as a failure. The resulting foxhole solidarity discourages wonks from calling out fellow travelers’ weakest work. Methodological criticisms are roughly equally effective regardless of their merit, because few journalists, pols, and voters can evaluate their strength, and even fewer actually will.
Knowledgeable wonks and writers should identify genuinely weak research, but we should be equally ready to praise methodologically sound work even when its findings and prescriptions are unwelcome. When ostensible technical fault-finding is actually sublimated ideological frustration, it threatens to reduce research quality at think tanks by debasing the value of technical criticism. Good research is expensive and difficult to produce, while the bad stuff is relatively cheap and easy. If no one signals the objective difference, bad research will inevitably crowd out the good. Perhaps a consortium devoted to recognizing the best research at each participating think tank could begin to realign incentives towards producing high-quality work.