Posted on Leave a comment

Another piece of the evaluation puzzle: Why do experiments make people unhappy?

The more time you spend around grants, grant writing, nonprofits, public agencies, and funders, the more apparent it becomes that the “evaluation” section of most proposals is only barely separate in genre from mythology and folktales, yet most grant RFPs include requests for evaluations that are, if not outright bogus, then at least improbable—they’re not going to happen in the real world. We’ve written quite a bit on this subject, for two reasons: one is my own intellectual curiosity, but the second is for clients who worry that funders want a real-deal, full-on, intellectually and epistemologically rigorous evaluation (hint: they don’t).

That’s the wind-up to “Why Do Experiments Make People Uneasy?“, Alex Tabarrok’s post on a paper about how “Meyer et al. show in a series of 16 tests that unease with experiments is replicable and general.” Tabarrok calls the paper “important and sad,” and I agree, but the paper also reveals an important (and previously implicit) point about evaluation proposal sections for nonprofit and public agencies: funders don’t care about real evaluations because a real evaluation will probably make the applicant, the funder, and the general public uneasy. Not only do they make people uneasy, but most people don’t even understand how a real evaluation works in a human-services organization, how to collect data, what a randomized controlled trial is, and so on.

There’s an analogous situation in medicine; I’ve spent a lot of time around doctors who are friends, and I’d love to tell some specific stories,* but I’ll say that while everyone is nominally in favor of “evidence-based medicine” as an abstract idea, most of those who superficially favor it don’t really understand what it means, how to do it, or how to make major changes based on evidence. It’s often an empty buzzword, like “best practices” or “patient-centered care.”

In many nonprofit and public agencies, evaluations and effectiveness are the same: everyone putatively believes in them, but almost no one understands them or wants real evaluations conducted. Plus, beyond that epistemic problem, even if evaluations are effective in a given circumstance (they’re usually not), they don’t necessarily transfer. If you’re curious about why, Experimental Conversations: Perspectives on Randomized Trials in Development Economics is a good place to start—and this is the book least likely to be read, out of all the books I’ve ever recommended here. Normal people like reading 50 Shades of Grey and The Name of the Rose, not Experimental Conversations.

In the meantime, some funders have gotten word about RCTs. For example, the Department of Justice’s (DOJ) Bureau of Justice Assistance’s (BJA) Second Chance Act RFPs have bonus points in them for RCTs. I’ll be astounded if more than a handful of applicants even attempt a real RCT—for one thing, there’s not enough money available to conduct a rigorous RCT, which typically requires paying the control group to follow up for long-term tracking. Whoever put the RCT in this RFP probably wasn’t thinking about that real-world issue.

It’s easy to imagine a world in which donors and funders demand real, true, and rigorous evaluations. But they don’t. Donors mostly want to feel warm fuzzies and the status that comes from being fawned over—and I approve those things too, by the way, as they make the world go round. Government funders mostly want to make congress feel good, while cultivating an aura of sanctity and kindness. The number of funders who will make nonprofit funding contingent on true evaluations is small, and the number willing to pay for true evaluations is smaller still. And that’s why we get the system we get. The mistake some nonprofits make is thinking that the evaluation sections of proposals are for real. They’re not. They’re almost pure proposal world.


* The stories are juicy and also not flattering to some of the residency and department heads involved.

Posted on Leave a comment

Links: Housing, grant size, the perils of EMRs, the nature of energy, addiction and treatment, and more!

* Death by a thousand clicks: How electronic medical record (EMR) systems went wrong. We’ve written so many proposals involving EMR systems, and yet it seems they’ve had little if any positive impact on the overall landscape, in terms of health or cost.

* “California Has a Housing Crisis. The Answer Is More Housing.” One of these obvious things, yet here we are.

* “When It Comes To Applying for Grants, Size Doesn’t Matter (Usually).”

* “A $20,243 bike crash: Zuckerberg hospital’s aggressive tactics leave patients with big bills. I spent a year writing about ER bills. Zuckerberg San Francisco General has the most surprising billing practices I’ve seen.” Remember how we wrote about the need for price transparency? This is another specific instance of that general point.

* Waymo’s CEO says autonomous cars “will always have constraints.” They are not a panacea for urban transit and are not going to be here in the next five years, and they will likely be weather-dependent.

* Is fusion power much closer to becoming reality than is commonly anticipated? If so, it will solve or substantially ameliorate the world’s energy problems, along with the geopolitical conflicts fueled by the world’s desire for oil.

* “Firms Learn That as They Help Charities, They Also Help Their Brands.” This is firmly “dog bites man” story instead of a “man bites dog” story, but there it is.

* “California will sue Huntington Beach over blocked homebuilding.” Good news.

* “Most People With Addiction Grow Out of It,” something not widely appreciated in the larger culture and a factoid we never include in the many SUD/OUD treatment proposals we write.

* Public Education’s Dirty Secret. Congruent with my experiences.

* “Is the Revolution of 3D-Printed Building Getting Closer?” Let’s hope so, as that would likely substantially decrease construction costs.

* Japanese urbanism and its application to the Anglo-World.

* “Climeworks: The Tiny Swiss Company That Thinks It Can Help Stop Climate Change.” Not just the usual.

* From Literature to Web Development: My first 6 weeks at Lambda School.

* * “A Radically Moderate Answer to Climate Change.” You may be getting tired of reading about nuclear power, yet we still seem as a culture not to be paying attention to it. See also “Nuclear goes retro — with a much greener outlook.”

* “This is Roquette Science: How computerized arugula (aka roquette) farms take over the world.”

* How to Create Reality: “So a funny thing happened on Twitter this week, which almost changed the world a little bit. Someone sent me a beautiful 3-D mockup of a fictional, car-free city of 50,000 people, set in the scenic nook of land* between Boulder, Colorado and Longmont, where I live.”

* “Science, Small Groups, and Stochasticity.” In short, we are doing the structure of science wrong.

* “The corporations devouring American colleges.” Colleges are businesses with extremely good PR and marketing arms.

* “The Streets Were Never Free. Congestion Pricing Finally Makes That Plain..” Seems obvious to me.

* “The antibiotics industry is broken—but there’s a fix.”

* “The 2008 financial crisis completely changed what majors students choose.” How could it not?

* “Lambda, an online school, wants to teach nursing.” Good. Competing with existing schools is a feature, not a bug. See also that other link about Lambda School, above.

* Most of America’s Rural Areas Are Doomed to Decline. Basically, agriculture now accounts for perhaps 2% of the workforce; manufacturing accounts for less than 15% of the workforce, and even as manufacturing has increased in value produced, it hasn’t much increased in jobs.

* “Considerations On Cost Disease‘s” money shot:

So, to summarize: in the past fifty years, education costs have doubled, college costs have dectupled, health insurance costs have dectupled, subway costs have at least dectupled, and housing costs have increased by about fifty percent. US health care costs about four times as much as equivalent health care in other First World countries; US subways cost about eight times as much as equivalent subways in other First World countries.

I worry that people don’t appreciate how weird this is.