TANF programs want to improve services for families to help lift them out of poverty,
but it’s hard to tell what will actually work for them.
Too often, those running programs simply don’t have the time or the resources for evaluations
or to sift through the existing research.
But what if there was a way to use the data, programs already collect, to provide timely
evidence about what works to improve services for families?
There is, and it’s called Rapid Cycle Evaluation.
Meet Pam.
She runs a TANF program in a mid-sized urban county, and knows from experience that clients
that attend orientation are more likely to enroll in the program and receive the help
they need.
But, only half of eligible families are showing up for orientation.
If she can increase attendance at orientation, she’s more likely to improve the lives of
the families she has spent her career trying to help.
She’s heard that simple “nudges”—like mailing a reminder postcard-- can encourage
participation and is curious if there’s something similar she could do to get more
people to attend the orientation and enroll in the program.
First, Pam and her team spent a bit of time exploring why only half the people were going
to orientation.
They found the enrollment paperwork had become long and confusing, making it hard for clients
to remember the important next step of coming to orientation.
Could something as simple as providing appointment reminder cards make a difference?
Even small changes like this come with high stakes.
She could buy materials, train staff, and roll out her idea to the hundreds of people
she serves each month, but what if the cards don’t work?
Pam can’t afford to waste those resources and that time without actually improving orientation
attendance.
That’s where Rapid Cycle Evaluation can help.
She can quickly test her idea using data she already collects, before rolling it out to
the whole program.
First, Pam designed a reminder card and piloted it by asking a few case managers to use it
with a few clients, and to give her feedback on how it worked.
Based on feedback, they updated the design and included the address on the card.
Then, they developed a detailed plan for further testing.
A small, simple, low-cost intervention like a reminder card won’t address all the reasons
why clients struggle to make their orientation sessions, so she knew she needed to define
success before going any further.
She determined that improving orientation attendance by 10 percentage points was a meaningful
improvement.
An evaluation partner looked at previous research and told her this was a reasonable outcome.
They knew what they wanted to do, and how they would measure success, but how would
they know if it was the cards making the difference?
Randomly assigning some participants to receive the reminder card, and some to get the business-as-usual
paperwork would allow Pam to compare the orientation attendance rates between the groups and determine
whether the new strategy was working.
Pam and her evaluation partner looked at the number of clients enrolling in the program
each week and determined it would take about 12 weeks to tell whether the strategy was
paying off.
Throughout the testing period, Pam talked with staff and clients to understand and document
how the cards were working.
After 12 weeks, they compared the average attendance rate of clients who received the
new card with the average attendance rate of those that didn’t and found a difference
of 8 percentage points—which didn’t quite meet her benchmark.
Based on feedback from staff and clients, Pam updated the design of the cards, added
a phone number to reschedule orientation appointments and re-tested the improved version.
After the second test, Pam and her evaluation partner found a difference of 13 percentage points.
The results exceeded her definition of success of 10 percentage points and Pam decided to
scale up the reminder card program-wide.
Based on her current numbers, that 13 percentage point improvement in orientation attendance
means Pam can expect about 150 more families each year to enroll in the program and start
on their path to economic security.
Not bad for a simple reminder.
For a reminder about how Rapid Cycle Evaluation can help you, plan a strategy that addresses
the problem and is meaningful enough to plausibly have an effect.
Decide how you will measure success.
Implement the strategy and monitor implementation and review/revise and use evidence to inform
decision-making, visit our website at the link shown here.