This article was written for and published in The Chronicle of Philanthropy on June 25th.
By Paul VanDeCarr
The “How Do We Know” web site uses garden tools as metaphors to provide fresh language for exploring media impact. Shovels, for example, represent investigative stories that dig for the truth.
For many storytellers — even advocacy storytellers — there is no more loathsome question than, “How are you going to measure your impact?”
And yet, as most any filmmaker, writer, or photographer working for the public good will tell you, that is a hard question to answer.
Grant makers need evidence that they’ve spent the foundation’s resources well. And, truth be told, the storytellers themselves want to know if what they’re doing is doing any good. After all, most people who tell stories about social issues want to promote change.
And yet the whole idea of measuring results clashes with the creative impulse behind storytelling.
The folks at Active Voice Lab — a San Francisco-based group that helps people use stories and culture to advance social change — have artfully reframed this question to get at something more essential: “How do we know if we’re making a difference?”
And that’s the topic – and the name – of a new beta website the group has launched.
Deciding whether to measure
The website includes frank perspectives on impact from grant makers, creative people, and others.
One filmmaker says, “It’s the funders who want evaluation. Period.” Another takes a contrary view: “Why not measure to see if we’re right, to learn more about what’s working and what’s not. Bring it on.”
I understand both perspectives.
Grant reporting can be a drag, especially when you’re just saying that you did what you said you’d do — and even more so if you’re not sure if anyone will actually read your report. And yet assessing a project can be useful in the same way that writing a grant proposal is: It forces you to articulate what you want to accomplish and why.
Active Voice Lab director Ellen Schneider says media-makers typically have one or more of the following motivations in measuring their impact: pleasing their donors, proving they did what was promised, demonstrating the value of their stories, affirming their creative contribution to the cause, improving their own work, or building the field as a whole.
Ms. Schneider says she’d like to see storytellers worrying less about pleasing and proving and focus more on the higher-level aims.
“But that will take time,” she says. “And a nudge here and there.”
If you have an idea of what you want to learn and why, evaluation can help. But it should be paired with a strategy for making a difference. Otherwise, it’s a little like telling only the end of a story: You say where you end up but not the path you took to get there.
But if you’re not under pressure to evaluate, or won’t learn anything from it — don’t bother.
That’s the gist of a 2014 article from the Stanford Social Innovation Review. The authors rightly add that “impact evaluations should be comparative”: They should verify whether a certain intervention worked better or worse than another.
The problem is that the dynamics of social change are so complex that an accurate comparison is nearly impossible in many cases. Did this film project contribute to a change in public opinion more than that community-organizing campaign or this other white paper by a think tank? Who knows? But that doesn’t mean we can only throw our hands up in the air.
Identify what you’re measuring
“There’s a lot we can measure when it comes to a story’s contribution to social change,” Ms. Schneider says.
But before getting too deep into the weeds, she suggests asking a basic question: “What do we hope to accomplish with this story, and is that outcome measureable?”
Specific goals like “recruiting new members for a local coalition” are easier to track than diffuse ones such as “changing the way people treat each other after a film screening,” Ms. Schneider says.
Go too narrow with what you’re measuring, and you risk showing only what’s self-evident. Go too broad and you risk trying to prove too much. The sweet spot is where social impact overlaps with your desire and ability to evaluate — say, by doing focus groups or surveys or gathering statistics on donations or new members.
The “How Do We Know” website has a set of “horticulture” garden tools that serve as metaphors to provide fresh language for exploring media impact. “Shovels” represent investigative stories that dig for the truth, “rakes” scratch the surface to bring together people from different perspectives, and “wheelbarrows” transport audiences without providing simple solutions.
Determine how to measure
Once you’ve figured out whether you want to measure and what you want to learn, it’s time to get into the nitty-gritty: How are you doing to go about it?
Among the many excellent listings on the “How Do We Know” resource page, a few of my favorites name specific tactics for measuring change.
The Fledgling Fund’s 2009 paper on media impact reminds readers to have a timeline for their evaluation plan and dedicate sufficient resources to it.
A 2014 paper by the Center for Media & Social Impact outlines the main evaluation methods, like surveys, focus groups, and too-often-forgotten experiments. (These are comparative studies in miniature—examining audience response to one piece of media versus another. For example, remember how the 2012 Obama campaign tested the response rate of various email subject lines? One of the most successful was simply “Hey.”)
Similarly, as I suggest in the Working Narratives guide on “Storytelling and Social Change,” organizations might pose a strategy question upfront that they’ll address over the course of a project. What kinds of stories raise the most money in fundraising appeals? What types of activities most empower “citizen journalists”? What media are most effective in communicating with target audiences? These sorts of questions generate knowledge for the organization, its donors, and other groups.
And Animating Democracy, a project of Americans for the Arts, has an outstanding page on social-impact indicators, to assess changes in such areas as people’s knowledge, attitudes, actions, and in policies. (It’s part of a broader set of a resources on the group’s website.)
Put yourself in context
“A grant maker once asked me a great question,” Ms. Schneider says. “‘Why do some filmmakers take credit for having impact when organizers have been working on these issues over the long haul?’ That was over 10 years ago, but it inspired us to look at the unique contributions that only stories can make.”
“I can’t think of a social-justice movement that doesn’t need stories, just as they need organizers, money, research, leadership. But stories alone are just characters, settings, plot, and so on. They’re powerful as part of an ‘ecosystem of change,’ ” Ms. Schneider concludes — a big picture that includes all the other things in a social-change movement.
The work of social change can feel abstract, especially if you sit at a desk much of the day or even if you’re out gathering and telling stories but aren’t sure who sees them or if they care. You may wonder if your dedication to your cause is worth anything. You may despair of the possibilities for change, when the forces of apathy and injustice seem so entrenched.
It’s in those moments especially when the question of “How do we know” matters most, for it is a prompt to stay connected to the work itself and other people doing it.
Paul VanDeCarr writes once each month about some of the best nonprofit storytelling and what others can learn from it. Readers can submit examples for consideration in this feature via this online form or e-mail Mr. VanDeCarr at paul@workingnarratives.org.
Mr. VanDeCarr is the managing director of Working Narratives, an organization that collaborates with advocates, artists, policy groups, media-makers, and others to “change the story” on the big social-justice issues of our time. He is also the author of “Storytelling and Social Change: a Strategy Guide for Grantmakers,” and is working on a second edition to be released this year for nonprofits, advocates, and storytellers.
