Monthly Archives: October 2008

All’s Well That Ends Well: A Tale of Hope on the Grant Writing Trail

All’s Well That Ends Well is one of Shakespeare’s “problem comedies” that may actually be a tragedy, and it comes to mind as an apropos title for a comedic tale that illustrates one of the many odd aspects of grant writing: why there is little reason to read comments provided by reviewers regarding an unfunded federal proposal*. Such comments, which may be mildly amusing or maddeningly frustrating to read, are usually useless in terms of improving the proposal for another submission. A long-time client’s experience demonstrates this.

Faithful readers will have followed Jake’s two-part deconstruction of the wondrous CBAE RFP, “What to do When You Still Must Fight Through a Poorly Organized RFP: Part II of a Case Study On the Community-Based Abstinence Education Program RFP.” This RFP  has been more or less the same for the last several years and pretty much sums up everything that is wrong with most federal RFP processes. Consequently, when a client called last week to discuss a possible new assignment, I was tickled to learn that their agency received a CBAE grant this year.

We wrote their CBAE proposal last year, which was not funded. The client received very negative reviewer comments on the ’07 proposal. Reviewers hated it. My advice to the client at that time was to ignore the comments and submit the proposal again this year. They submitted a proposal this year, virtually unchanged from the one that was trashed, and they were funded.

The primary reason for not taking reviewer comments seriously is the nature of the people reviewing it. Any proposal is read at a point in time by a set of reviewers, who are likely reading other proposals submitted for the same competition and may or may not be interested in the task at hand.

For example, if the proposal is read by five peer reviewers brought to D.C. by DHHS, one may be hung over from bar hopping the night before in Georgetown, one may be anxious to meet their Aunt Martha for dinner, a third may be itching to get to the Air and Space Museum before it closes, and two might be vaguely interested in the review process. And, of the last two, one may have gotten a speeding ticket in your jurisdiction 20 years ago and hates the city. Or, the proposals could be read by federal zombies**, who are disinterested in everything placed before their noses other than donuts. In other words, one has no control over who reviews proposals and what kind of mood they might be in.

Also, just like with job interviews, in which the time of day may make all the difference, a proposal read first thing in the morning by bright-eyed reviewers will likely fare differently than one read by the same bleary group just before the cocktail hour. If that weren’t enough, the values of one reviewer one year might differ greatly from the values of another reviewer another year, and the priorities of the agency might shift slightly from year to year, meaning that a proposal that they hate one year they might love the next. If you submit a complete, well-written, and technically correct proposal, you’ve done as much as you can, and the vagaries of the reviewer can doom or save your application.

It’s pointless to agonize over grant reviewer comments. When our clients send them to us, we look at them briefly to see if there was something obviously wrong with the proposal, but usually there are either just points assigned to various sections or cryptic comments like, “collaboration not demonstrated,” even though there were ten letters of collaboration attached, a list of partners included in the narrative, etc. Sometimes reviewer comments can be either unintentionally hilarious or tragic, depending on your point of view, which brings us back to the Bard.

A case in point: About eight years ago, we wrote a proposal to the Administration for Native Americans (ANA) Social and Economic Development Strategies (SEDS) program on behalf of an Alaskan Native village north of the Arctic Circle for a social development project. The proposal was not funded and the client faxed the review comments to us. Imagine our surprise when the comments talked about the poorly developed project concept of trying to attract tourists to a remote desert area in Arizona! Clearly, the reviewers mixed up at least two proposals and associated the wrong comments with the proposal we had written. We advised our clients to contact ANA, which, in the true spirit of helping vulnerable Alaskan Natives, refused to accept an appeal for what was obviously a major error. Perhaps a better Shakespeare title for this post would be Love’s Labour’s Lost.

* As the King says, perhaps ironically, at the end of the play, “All yet seems well, and if it end so meet [fittingly], / The bitter past, more welcome is the sweet.”

** While not generally a fan of zombie movies, the most fun iteration in recent years is certainly Shaun of the Dead in which the eponymous hero has trouble at first distinguishing his slacker friends from zombies rising from the grave like grant reviewers emerging from the sub-basement at DHHS after a long day of reading proposals and stampeding to the closest Happy Hour.

What to do When You Still Must Fight Through a Poorly Organized RFP: Part II of a Case Study On the Community-Based Abstinence Education Program RFP

This is the second of two posts; the first appears here.

In addition to being unsupported by the research demanded by the program, as described in this post, the Community-Based Abstinence Education Program (CBAE) RFP is also poorly organized. It separates concepts and ideas that belong together for no apparent reason. This is most evident in its pursuit of the “A – H criteria” on page 2, the evaluation criteria on pages 5 – 8, and the “13 themes” on page 11. Each corresponds to parts of what appear to be the narrative on pages 52, 51, and 52, respectively. This raises the obvious question of why the RFP writers didn’t simply integrate the A – H, evaluation, and 13 themes criteria in a single spot, rather than referring endlessly back and forth between them. In addition, since the evaluation section on pages 5 – 8 tells you what you have to do for an evaluation, it might’ve been wise to just tell the applicant how to run the evaluation, rather than have the applicant pretend they’re going to design their own evaluation.

Still, this raises another important grant writing point, which is that you should, whenever possible, take elements from the RFP and regurgitate them in your proposal, like a mother bird vomiting up dinner for her young. The easiest way to conform to the RFP and program requirements is to do exactly what the requirements tell you to do in language that mirrors but does not copy the RFP. Page 7 of CBAE says:

In addition, successful applicants will be required to administer a 10-item survey to all participants as a pre-test and post-test when appropriate (e.g., when students are participating in a curriculum- based program). They will also be required to administer a post-post test to a representative sample of participants six months and/or 12 months after completion of the abstinence education program.

Such a passage can be easily rewritten as part of the evaluation section, and it’s an easy way to gain points. When RFPs give you lemons, make lemonade, and this RFP certainly gives a lot of lemons.

Notice, however, that three paragraphs above I said “what appears to be the narrative.” That’s because two sections of the RFP could arguably be the narrative: one that starts on page 41 under the header “Part II GENERAL INSTRUCTIONS FOR PREPARING A FULL PROJECT DESCRIPTION,” and another that starts on page 48 under “EVALUATION CRITERIA.” But the one that says evaluation criteria doesn’t refer to the evaluation you’re going to run—it refers instead to how the funding agency is going to decide whether you should be funded. In other, recursive, words, they’re going to use their evaluation criteria to evaluate your evaluation criteria. Simple, right? Welcome to grant writing.

The question for the grant writer is how to choose whether to write to the narrative directions on page 41 or page 48. When in doubt, try to mimic as closely as possible whatever checklist the reviewers will use. That makes it easy for them to see that you’ve hit all the elements, and it’s more important to hit the right elements, even poorly, than hit the wrong elements well. This is another RFP trap, and a variation on some of the traps described in the linked post; I’d also bet it was constructed using the methods described in Inside the Sausage Factory and how the RFP Process leads to Confused Grant Writers.

The CBAE RFP has other traps:

*The required abstract has directions placed randomly on page 24. Miss those and your application could be thrown out. Why don’t they put the instructions with the rest of the narrative instruction? Beats me.

* Another items problem that isn’t a trap, exactly, but worth observing, is that ACF has apparently realized the drawbacks Isaac enumerated about Page 28 says: “Applicants may submit their applications in electronic format through (see Electronic Submission section) and/or in hard copy (see Section IV.3).”

* If those weren’t enough, the RFP also implies on page 5 one potential ordering for the evaluation. But it implies another on page 43, under the “General Instructions for Preparing a Full Project Description.” Yet page 28 says one should “[a]rrange all materials in the order listed in the Application Content section above.” Above, on page 25, we find yet another potential way of ordering the evaluation. We didn’t figure this out from the RFP, though—we called ACF to ask about the section order and a staffer tell us that our responses to the evaluation criteria in the RFP could go in any order. This alleviated the problem of the order in the narrative section, which originally asked for the needs assessment, then the evaluation, then the approach, otherwise known as “what you’ll do.” But how are you supposed to describe how you’re going to conduct the evaluation if you haven’t described what you’re going to evaluate? Not clearly, anyway. The mixed up order is yet another example of the carelessness of whoever wrote this RFP.

Finally, the header that appears on page 10 says: “Successful Abstinence Education Curriculum.” This is another ironic title, since the RFP is filled with references to trying to evaluate the effectiveness of the program. Yet the header presumes that abstinence education will be successful, and the content it lists shows what elements have to be included. But if we don’t know whether programs with these elements work, how can we know that it will be successful? The answer is, of course, that we can’t. Isaac wrote in Self-Efficacy—Oops, There Goes Another Rubber Tree Plant about “those districts have ironically named “successful school guides” even as their students fail almost every ability test imaginable.” The same concept applies here: the RFP defines what kind of abstinence education will be “successful,” then tells you what to do, then expects you to tell them what you’re going to do, and then expects all this to be evaluated to figure out whether it’s successful. Follow all that? Neither do I, and I just wrote it. No wonder the bureaucrats who prepared this RFP confused themselves and applicants.

The organization of this RFP might be an example of the frustrations that come from abstinence—it’s a case of mental masturbation. The writers probably thought about a joke regarding masturbation from Woody Allen’s Annie Hall: at least it’s sex with someone I love.

EDIT, August 2012: In “Parents Just Don’t Understand: A sociologist says American moms and dads are in denial about their kids’ sexual lives,” Sinikka Elliott argues about comprehensive sex education and abstinence education:

One side is saying, ”Well, they need to abstain. That’s a surefire way that they’re gonna be safe,” and the other side is saying, “They’re not gonna abstain and so they need contraceptive information.” They were basing their argument on the same things: the teen pregnancy rates, the STI rates.

Neither focuses much on pleasure, as Elliott points out.

Perhaps the most hilarious thing about the CBAE RFP is how long virginity has been ineffectively pitched in Western society. Foucault and Barzun show that in part, and now Orwell does too in his Essays, this one dating from 1947:

We shall never be able to stamp out syphilis and gonorrhea until the stigma of sinfulness is removed from them. When full conscription was introduced in the 1914 – 1918 war it was discovered, if I remember rightly, that nearly half the population suffered from some form of venereal disease, and this frightened the authorities into taking a few precautions […]

You can’t deal with these diseases so long as they are thought of as visitations from God, in a totally different category from all other diseases […] And it is humbug to say that “clean living is the only remedy.” You are bound to have promiscuity and prostitution in a society like ours, where people mature sexually at about fifteen and are discouraged from marrying till they are in their twenties […] (1217-8).

The purpose in observing this is not to endorse Orwell’s perspective, but to point out, as Isaac often does, that the kind of debates that spur RFPs and social policy and all the rest are very seldom new ones, and the various remedies offered are equally ancient.

What to do When Research Indicates Your Approach is Unlikely to Succeed: Part I of a Case Study on the Community-Based Abstinence Education Program RFP

The Community Based Abstinence Education Program (CBAE) from the Administration on Children, Youth and Families is a complicated, confusing, and poorly designed RFP based on suspect premises. That makes it an excellent case study in how to deal with a variety of grant writing problems that relate to research, RFP construction, and your responses.

CBAE is simple: you’re supposed to provide abstinence and only abstinence education to teenagers. That means no talk about condoms and birth control being options. In some ways, CBAE is a counterpoint to the Title X Family Planning funding, which chiefly goes to safe-sex education and materials rather than abstinence education. Its premise is equally simple: if you’re going to have sex, use condoms and birth control. Congress chooses to fund both.

Were I more audacious regarding CBAE proposals, I’d have used George Orwell’s 1984 as a template for the programs, since almost everyone in the novel conforms to the numbing will of an all-powerful state and many belong to the “Junior Anti-Sex League,” complete with scarlet sashes. I hope someone turned in a CBAE application proposing scarlet sashes for all participants.

More on point, however, page two of the RFP says:

Pursuant to Section 510(b)(2) of Title V of the Social Security Act, the term “abstinence education,” for purposes of this program means an educational or motivational program that: […]

(B) Teaches abstinence from sexual activity outside marriage as the expected standard for all school age children

Who is enforcing this “expected standard?” Society in general? A particular person in society? But it gets better:

(D) Teaches that a mutually faithful monogamous relationship in the context of marriage is the expected standard of human sexual activity;

This requirement ignores decades of anthropological research into indigenous societies as well as plenty of research into our own society, which Mary Roach described in Bonk, Alfred Kinsey described using imperfect methods in his famous but flawed research in the 50’s, and that Foucault described in his History of Sexuality. It also ignores the sexuality of other cultures and even our own, as discussed in books like Conceiving Sexuality: Approaches to Sex Research in a Postmodern World, or, better yet, Culture, Society and Sexuality: A Reader, which describes the way societies and others build a social model of sex. Through the CBAE program, Congress is building one such model by asserting it is true and using “expected standard” language, without saying who is the “expecting” person or what is the “expecting” body. It’s an example of what Roger Shuy calls in Bureaucratic Language in Government and Business a term that “seems to be evasive,” as when insiders “use language to camouflage their message deliberately, particularly when trying to avoid saying something unpleasant or uncomfortable.” In this case, the evasion is the person upholding the supposed standard.

Furthermore, the abstinence conclusion isn’t well supported by the research that does exist, including research from previous years of the program, which is at best inconclusive. A Government Accountability Office report (warning: .pdf file) says things like, “While the extent to which federally funded abstinence-until-marriage education materials are inaccurate is not known, in the course of their reviews OPA [Office of Population Affairs] and some states reported that they have found inaccuracies in abstinence-until-marriage education materials. For example, one state official described an instance in which abstinence-until-marriage materials incorrectly suggested that HIV can pass through condoms because the latex used in condoms is porous.”

The one comprehensive study that has been conducted by a nonpartisan firm is called “Impacts of Four Title V, Section 510 Abstinence Education Programs” by Mathematica Public Research, which was spun off from the guys who brought us the Mathematica software. The study was prepared for DHHS itself, and it says such encouraging things as, “Findings indicate that youth in the program group were no more likely than control group youth to have abstained from sex and, among those who reported having had sex, they had similar numbers of sexual partners and had initiated sex at the same mean age.” The programs it studied are based around the same methods that the CBAE demands organizations use, all of which boil down to inculcating a culture of fear of sex outside of marriage. The social stigma the program recommends is based around STDs and whether you’ll get into college (although an editorial in the L.A. Times argues otherwise), and, to a lesser extent, altering peer norms. Still, even in Puritan times this was not entirely effective, as Bundling by Henry Stiles explains. The practice meant sleeping in the same bed with one’s clothes on, as a solution to the problems of inadequate heat and space. But, as Jacques Barzun says in From Dawn To Decadence: 500 Years of Western Cultural Life, “Experience showed the difficulty of restraint and […] the rule was made absolute that pregnancy after bundling imposed marriage […] So frequent was this occurrence that the church records repeatedly show the abbreviation FBM—fornication before marriage.”

There are counter-studies that purport to show abstinence education as effective, like this one from a crew that, not surprisingly, is selling abstinence education materials. But it, like most others, has little bon mots amid its intimidating numbers and verbose language like, “In addition, the high attrition rate limits our ability to generalize the findings to a higher-risk population” (strangely enough, the .pdf file is set to disallow copying and pasting, perhaps to discourage irate bloggers like myself). But the study doesn’t list the attrition rate, making it impossible to tell how severe the problem is. In addition, even if it did, the population selected might also suffer from cherry picking problems of various kinds: that is to say, organizations are more likely to serve the participants who are most likely to be receptive to services and, concomitantly, less likely to do things like have early sex. This is an easy and tempting way to make a program look good: only let the kids in who are likely to benefit. And it’s a hard problem to tease out in studies.

So be wary of dueling studies: if you don’t read these carefully, it’s easy to accept their validity, and even if you do read them carefully, it’s easy to nitpick. This is why peer review is so helpful in science and also part of the reason evaluations are so difficult. Furthermore, many of the studies, including Heritage’s, come from biased sources, a problem Megan McArdle writes about extensively in a non-abstinence-related context. (See her follow-up here). Most of you justifiably haven’t followed the blizzard of links I put up earlier or read the books I cited for good reason: who has the time to sift through all this stuff? No one, and even pseudoscience combined with anecdote like this article in New York Magazine has an opinion (hint: be wary of anyone whose title has the word “evolutionary” in it).

Given this research, which is hard to miss once you begin searching for information about the efficacy of abstinence instruction, how is a grant writer to create a logic model that, as page 44 says, should list “[a]ssumptions (e.g., beliefs about how the program will work and is supporting resources. Assumptions should be based on research, best practices, and experience)”? (emphasis added).

Two words: ignore research. And by “ignore research,” I mean any research that doesn’t support the assumptions underlying the RFP. If you want to be funded, you simply have to pretend “Impacts of Four Title V, Section 510 Abstinence Education Programs” or the GAO study don’t exist, and your proposal should be consistent with what the RFP claims, even if it’s wrong. This is, I suspect, one of the hardest things for novice grant writers to accept, which is that you’re not trying to be right in the sense of the scientific method of discerning the natural world through experimentation. You’re trying to be right in the Willie Stark sense of playing the game for the money. No matter how tempting it is to cite accurate research that contradicts the program, don’t, unless it’s to knock the research.

Remember too that the grant writer is to some extent also a mythmaker, which is a subject Isaac will address more fully in a future post. The vital thing to consider is that the mythology you need to create isn’t always the same as the reality on the ground. As in politics, the way events are portrayed are often different than how they actually are. David Broder wrote an article on the subject of inventing political narratives, which occasionally match reality; your job as a grant writer is inventing grant narratives. We hope these match reality more often than not. Sometimes the myth doesn’t, as in this application, and when that happens, you’re obligated to conform to the RFP’s mythology, even if it isn’t your own.

The second part of this post continues here.

Stay the Course: Don’t Change Horses (or Concepts) in the Middle of the Stream (or Proposal Writing)

Before you start writing a proposal, it’s good idea to understand the project concept and stick with this concept throughout the various drafts. If you don’t, the probability of creating an incomprehensible mess is high. In other words, it is a spectacularly bad idea to make big changes during the final stages of finishing the proposal.

This simple truth was reinforced during the recent HUD SuperNOFA season, when we experienced the grant equivalent of The China Syndrome. We were writing a proposal for one of the SuperNOFA programs on behalf of a public sector client for whom we had written the same proposal twice previously. Both those applications were funded. The program itself was and is very complex, and it requires close collaboration between city and county entities to meet HUD requirements.

Since we were familiar with our client’s past approach and our contact person did not correct us during our scoping session, we wrote the first and second drafts based on what we thought was the project concept. After she reviewed the second draft, I got a panicked phone call from our client telling me that she had decided to radically change the service delivery model, even though we were within three or four days of the deadline. I told her this was a very bad idea for all kinds of reasons, but she persisted—with predictably disastrous results. Although the proposal got submitted, it was a nail-biter and probably went in with lots of mistakes.

Here’s why . . .

Most proposals have intertwined narratives in which program elements are threaded throughout the document. In the example above, one major issue was a collaborator, and our client decided change its role at the last minute. While I made a yeoperson’s effort to find and correct all of these instances, it is likely that some were missed, meaning that the proposal probably went in with internal inconsistencies.

Such gaffes are not obvious if one has read the draft proposal a dozen times, but they will stand out like neon lights to a reviewer and lead to lowered points, if not outright laughter in the event that a group is reviewing the proposal. The particular change she made also affected the budget and budget narrative. For HUD proposals, there are up to four budget forms that must be completed, in addition to the budget narrative, so a last minute change may introduce budget errors. In federal grant reviewing, budgets are usually not scored, but inconsistencies between budget forms and the narrative can be a fast way to the exit.

The HUD example was a submission, causing another major problem. As faithful readers will recall from “ Lurches Into the 21st Century,” allows itself up to 48 hours to declare that an acceptable upload has been received. Thus, we always recommend that deadlines be moved up 2 – 3 days to allow for file corruption, server downtime, etc.

Our client had us making last minute changes to the proposal up to the night before the deadline, which meant the application kit file was not ready for upload until the morning of the due date. Of course, when she tried to upload the file, rejected it. It took all day and hours on the phone with tech support for her to resolve the problem. If she had not insisted on major last minute changes, the upload attempt would have been done two days earlier and the problem resolved without drama. Since the proposal request was $3 million, there was a lot on the line.

The moral of this sad tale: get the project concept straight at the start and resist the urge to change directions near the end of the process. Successful grant writers must be single-minded to produce a technically correct proposal on time. Radical alterations to the proposal concept at the last minute is a variation on “The Perils of Perfectionism.” Grant writers should work on a proposal until they’ve worked on it enough, then button it up, submit it in time to easily meet the deadline and retire to a nearby bar for a cocktail or two. Try a Negroni which is an excellent way to get over a stressful proposal writing experience.

EDIT: You can find a follow-up describing how this experience ends in “Now It’s Time for the Rest of the Story.”