…

I run a data analytics firm that utilizes technology to help organizations evaluate their effectiveness and improve results. Given the hype around data in the social sector, it’s not a bad time to be a nerd for social good.

But amidst all this hope and promise, I’m afraid our collective enthusiasm for hackathons and data visualizations have allowed us to stray from the following two simple rules:

1. Technology should help, not hurt
2. People make decisions, not data

If evaluative metrics are to become a core component of the social sector’s work, frontline organizations need to have the freedom to use the technologies that work for them, free from funder mandates. More important, the sector needs to invest in raising the collective data literacy of all social sector workers.

Indeed, data analytics should not be the realm of us nerds alone. Instead, a basic understanding of evaluative principles should be a prerequisite for entering the field.

Teched out

Every organization should have a data collection system, as has long been argued by nonprofit consultant David Hunter. While there are organizations that don’t have any formalized mechanism to capture outcomes data, the organizations I work with typically enter their outcomes data into multiple databases, one for each funder while also maintaining a database for themselves.

This is a ludicrous practice. I understand the desire for funding organizations to receive raw data from their grantees, and whole heartedly support this objective. But grantees should be free to use the data collection system of their choosing, instead of being forced to duplicate entries into any number of proprietary databases. Not only does this practice waste frontline organizations’ time, it also conflates evaluation with compliance reporting, leaving organizations with little appetite to explore data for program improvement.

Funding entities have a significant opportunity to remove an unnecessary pain from the evaluation process by backing away from forcing grantees to enter information into proprietary systems and instead supporting data interoperability via open standards.

Building data repositories is a fairly trivial undertaking, which I believe is why so many funders have opted to develop their own. Of course, the real cost of these systems is not the financial burden of providing a database to grantees, but rather the social loss of grantees’ time and energy spent on duplicative data entry.

Organizational learning

Where and how to store data is not a terribly interesting problem. The more interesting question is what to do with outcomes metrics, and how to use evaluative techniques to improve social impact.

While fancy visualizations and infographics have raised the visibility of data in the social sector – these efforts have done little to raise our collective evaluative IQ.

In my work, I help social sector organizations use their outcomes metrics to develop predictive models that not only inform how an organization is doing, but provide insight into how their interventions can improve. Whatever the analytical focus of each of my engagements, the subtext is always a focus on cultivating organizational competence around evaluative inquiry.

Of course, learning doesn’t happen in punctuated points. There is a reason students are assigned homework and take multiple quizzes in an academic term prior to taking a final exam. Homework and quizzes improve learning, while the final exam is an opportunity to demonstrate knowledge.

But to date, we as a sector tend to treat evaluation as the equivalent of administering a final exam to a room full of students who have not attended class. Funders require annual reports demonstrating success and evaluation consultants swoop in to write one-off reports that effectively assign organizations letter grades. This is the wrong way to approach evaluation.

A better approach is to invest in developing social sector workers’ ability to understand what their data says, what questions evaluative inquiry can inform, and appreciation for the limits of statistical analysis.

Ultimately, understanding evaluative principles is everyone’s responsibility. Just as everyone in the medical profession knows what a carotid artery is, so too should everyone in the social sector know what a counterfactual is. To get there, we need to support organizations in choosing one data collection scheme that works for them, and reject the anti-intellectualisms surrounding evaluation that favors pretty graphs over mastering terminology.

Steve Wright, Vice President, Poverty Tools and Insights at Grameen Foundation joins the discussion we raised yesterday on beneficiary insight. There, we addressed an article published in Alliance Magazine by David Bonbright, expressing the concern that Markets For Good may go wrong, arguing that “beneficiaries are not part of the enterprise in a direct or meaningful sense.” See the links above to learn how we have considered this critical topic from the outset. Below, Steve joins in… and joins the themes: beneficiary insight and our current theme, evaluation. He addresses a few difficult questions (and leaves us with more) regarding data-collection methodologies and what they should mean for beneficiaries.

…

There is a very tired and self-congratulatory axiom from the for-profit community that declares “nonprofits should act more like businesses.” Now that business-like includes illustrative examples of LIBOR rigging, mortgage securities collusion and too-big-to-fail we have the opportunity to throw out the social vs. private distinction and focus on value creation. This is the foundation of Markets for Good as I understand it. How can we best provide value to customers? The difficulty here is that value is subjective and the goal is to provide value as realized by the customer. Fortunately, over the last few years we have seen two specific powerful innovations that provide a methodology for any business to be more focused on the value that is realized by their customers – Human Centered Design and the Lean Startup Methodology. Both of these have been practiced to great effect by every stripe of organization. I bring this up to make the very specific and simple point that these are the methodologies that are essential to ensuring not just the inclusion but the increased agency of the beneficiaries of Markets for Good.

Let’s take the very specific example of poverty, the poor and organizations that intend to alleviate poverty. Whether it’s fair trade, social enterprise, microfinance, a health clinic, or an agricultural co-op, any business that intends to help the poor must be able to measure poverty. Without knowing if your customers are objectively poor you cannot know if you have effectively designed your products and services, or optimized your business processes, to provide the greatest possible value to your customers. The Progress out of Poverty Index® (PPI®) – managed by the Grameen Foundation – is a survey and scoring system that uses specific financial expenditures as indicators to accurately estimate a household’s level of poverty as compared to an internationally recognized poverty line. By using the PPI an organization can answer question like:

What percentage of my customers are below the National Poverty Line?

Are my customers moving relative to poverty over time?

Exactly how are my poorest customers interacting with my business?

With the PPI any business or organization that works with the poor can easily embed poverty measurement into its operations. Measuring poverty can provide any organization that intends to alleviate poverty the data they need to manage their business. CARD Bank in the Philippines implemented the PPI and were able create a dramatic change in the adoption of voluntary savings accounts by their poorest customers.

“We are excited about the additional information the PPI gives us on our customers. Because of the richer analysis we are now able to do, we can be confident that we are designing savings products that suit the needs of the poor and reaching our overall poverty targets.” said Dolores M. Torres, president and CEO of CARD Bank.

Data Should “Fall Out” of your Social Business
If your business focuses on providing value to be realized by your customer then the data that falls out of the act of managing your business becomes a reasonable proxy for the data you need to assess the performance of your business. This is the principle of manage what matters.It is critical to intentionally manage your business in a way that the activities you engage in are explicitly designed to provide the value that is realized by your customers. Then, as you work with your customers to understand how well they are served by your business, any change you make to your products, services or processes are also changes made to more effectively advance your mission. Measurement and management, in a customer focused business, are integral.

A great example of an organization that is doing this well is VisionSpring. For over a decade, Vision Spring has been fighting poverty by selling affordable eyeglasses to the poor. Historically it has been too difficult or too expensive for them to get good data about their customers. While this data is clearly valuable, its value to VisionSpring must always be balanced against the cost of collecting the data and therefore the cost of running the business. It is a very straight forward ROI calculation. Like many social enterprises in the developing world, VisionSpring sells its products in difficult and emerging markets where the cost of doing business is high especially when the intent is to reach underserved populations. Using Grameen Foundation’s Taro Works product to collect customer information at the point of sale, VisionSpring now has access to data that gives them the insight they need to continuously improve their products and services.

This chart shows the likelihood that a customer is below the National Poverty Line for each of 9 different regions where VisionSpring sells reading glasses.

In this chart VisionSpring for the first time has access to their customers motivations in El Salvador.

“Would you Refer” metrics are known to be very good indicators of customer satisfaction.

Markets for Good

The thesis of this post and the goals of the organizations mentioned above is that a good business – and a good market – is one that strives to provide value that is realized by customers. This is the obvious and ultimate goal. It is also true that there is a broad group of stakeholders beyond the social enterprise and its customers that is integral to the work of markets for good and essential if we are to approach solutions to some of the world’s most intractable problems. The great challenge that is before us is to figure out how to align the incentives. How can competitors focusing on market solutions to clean water collaborate enough to ensure that their work is additive if not exponential? How can funders and investors align their data needs with those of the interventions that they are funding so that both are taking and the same risks and gaining the same rewards? Maybe most problematically, how can we co-opt profit maximizing markets to scale this disruptive innovation of delivering value as realized by the customer?

Markets for Good aims to foster a robust conversation about how people in the social sector can better generate, share, and use information to make better decisions and improve lives.

In a critique posted to Alliance magazine’s website last week, our friend and Markets for Good contributor David Bonbright shared his concerns that our efforts may go wrong, arguing that “beneficiaries are not part of the enterprise in a direct or meaningful sense.” In fact, the challenges and benefits regarding information from people in communities – beneficiaries, constituents, citizens—have been a constant theme on Markets for Good since we launched in October.

David urges Markets for Good to incorporate a “deliberative element” into the work, writing: “to realize ‘greatest impact’, the social change information infrastructure would need to begin and end with the primary constituents of social change – those who are meant to benefit.” I completely agree with this goal, which is indeed a key element to the vision, described on the site and in the white paper released in fall, 2014. [See page 2, first paragraph.] But David goes on to write that it “seems that Markets For Good is settling on a neutered understanding of beneficiaries as consumers of information about service availability and eligibility requirements. For the beneficiaries, the information flow is one-way, top-down.”

Nothing could be further from the truth. In fact, we’ve always imagined beneficiaries – as well as funders and social enterprises – as both contributing information about needs and resources and using similar information contributed by others. Just as with the web itself, users in this complex system are connected and networked and play different roles at different times.

It would be instructive, at this point, to review this compilation of the rich and diverse content we’ve published on the topic.

In the spirit of openness and deliberative intent which guides the Markets for Good efforts, we are pleased to host further conversation about this important issue. To that end, we’re re-posting (with permission) David’s full critique below, as well as a response in the comments by Jacob Harold of GuideStar.

You can add to this dialogue in the comments below. We are always looking for new perspectives as well as great resources and initiatives to highlight on Markets for Good; so if you have suggestions, just send them to info [at] marketsforgood.org.

…

FULL TEXT: Original Article in Alliance Magazine

Why Markets For Good may go wrong

Back in June 2010 I was one of 46 self-described ‘top visionaries and practitioners’ in the broadly defined ‘markets for giving space’ who came together to explore how to create greater social impact through the philanthropic ecosystem. The debates were intense, but to my surprise and delight at the end of three days, we emerged with a tantalizingly transformative vision for our social change information infrastructure.

This propitious beginning of Markets For Good placed the ‘beneficiaries’ at the heart of the vision [1]. We left the meeting with a shared understanding that the central challenge was not technical, but political [2]. We wanted an information marketplace that rendered the most marginal voices in the ecosystem the most important ones. The beneficiaries’ thought bubble in the diagram we produced read: ‘OMG! We are leading change, seeing change, and we ourselves are changing.’

Three years on, a few drafts of a concept paper, and nine months of an ‘online campaign’ later, Markets For Good seems to have re-conceptualized the core problem in more technical and strategic terms [3]. We have drifted from a transformative vision that put those who are meant to benefit from social change at the centre to one in which these primary constituents have become just another cog in the machine rather than the flywheel that determines the way the whole system works.

The reason for this, I believe, is that the beneficiaries are not part of the enterprise in a direct or meaningful sense. This article will explore this inconvenient truth about Markets For Good, while making the case that it would be a far more exciting and impactful enterprise were it to open its doors to them.

Before going further with a critique, I would like to indicate that the problem does not lie in the stewardship of the enterprise. The core leadership – GlobalGiving, Liquidnet, Hewlett Foundation and more recently the Gates Foundation – have managed a process that has been open, inclusive, careful, and in every way embracing of diverse points of view. Website curator Eric Henderson has been intellectually supple and proactive in soliciting participation. No, the problem does not lie here.

To the contrary, a plurality of the invited blog contributions to the site, including my own, are cautionary in nature. There has been a whole season of pieces on the beneficiary feedback. One piece in particular – that of Mauricio Lim Miller of the Family Independence Initiative – tackles the central challenges head on. Maurice’s work models what it means to build a people-centred information infrastructure for social change: what information is most important; who collects the information; how the information is used; and who ultimately declares success. In some ways, Markets For Good needs to do little more than follow the path already charted by Mauricio and his colleagues at FII.

When does a good process tend to a sub-optimal outcome?

But instead of holding to the original vision of the beneficiaries as the core agents of change, its seems that Markets For Good is settling on a neutered understanding of beneficiaries as consumers of information about service availability and eligibility requirements. For the beneficiaries, the information flow is one-way, top-down.

This prompts the troubling question, why is an exemplary process tending towards the wrong result? If Markets For Good is in every visible way a well-intentioned enquiry seeking a clear result – better lives for those who most need it and solutions to our big societal problems – why is it tending towards technocratic investments in information collaboration, like taxonomies and coding and data interoperability, rather than honouring the more politically seasoned theory of change that occasioned its birth?

I think that the answer lies in the politics of the initiative, which are bounded by four constituent groups, and the inconvenient truth that the beneficiaries are not ‘on the bus’.

These four groups are:

• The funders
• The information intermediaries
• The non-profit service delivery and advocacy groups who have identified information as a critical factor in their work
• Finally, the rest of the non-profit sector [4]

Funders
The funders are the keystone species in this ecosystem – take them out and the system would immediately collapse. One of the dominant characteristics of this ecosystem is that there are very few funders that want to be part of it in any programmatic way. Those prepared to finance the information infrastructure ecosystem are pummelled with applications from far and wide and get a great view of the fragmentation and duplication out there. Their natural response is to push for better collaboration and coordination – which is enshrined in the original framing question for the whole enterprise.

Information intermediaries
The most prolific species in the ecosystem is the information intermediaries. Their rate of growth is limited by the available grant resource for which they compete. There are various sub-species of intermediary, including online giving markets, technology and software, those doing evaluation of some kind, those setting performance standards, constituency mobilizers, and other new philanthropy intermediaries who define their comparative advantage in some way by their take on information. Across all these groups are two major fault lines. One is technology-driven vs politically driven theories of change; the other is non-profit sector insider vs outsider status. Both these fault lines are epitomized in the yin and yang of two big players, GuideStar and Charity Navigator.

Non-profit service delivery and advocacy groups who have identified information as a critical factor in their work
The third and most transformative species in the ecosystem is made up of a relatively small number of non-profits who do direct service or advocacy but whose approach to system-changing has led them to incorporate an ‘information as power’ element in their strategies. The Family Independence Initiative referred to earlier is one example. LIFT is another.

The rest of the non-profit sector
A few members of the fourth species – the bulk of US non-profits – are lurking with intent on the edges of the website discussions. They keep a wary eye on Markets For Good in case it comes up with new, irrelevant and burdensome measurement and reporting imperatives. They flash their approval when one of the information intermediaries expresses their worries about Markets For Good, as in Laura Quinn’s excellent post identifying a long list of challenges and inconsistencies in the funder-driven demands for better information. Laura’s post concludes: ‘These organizations rely on funders to help them meet their missions, but sometimes the burdens put on them by the reporting requirements that come with that funding can make it more difficult for them to carry out their work.’ We don’t see the Markets For Good web analytics (note to Markets For Good leadership: this is something that could be shared), but one can see that this post prompted a record number of comments from non-profits, all praising it. The vast majority of non-profits are going to need both clear guidelines and explicit support to become generators of the kind of enhanced information world envisioned by the Markets For Good enterprise.

The inconvenient truth about Markets For Good

This brings us to the inconvenient truth about Markets For Good. With this polity, Markets For Good’s tendency will be – as seems to be working out – to settle for efficiency gains to the present system. Since the beneficiaries are not part of the Markets For Good ecosystem, except in name, it is unlikely that their centrality will be realized. What happened at the meeting in June 2010 was an aberration, not a harbinger.

The Markets For Good enterprise that was envisioned in June 2010 had two primary hypotheses coming in and three coming out. Some three years on, it has defaulted back to the two original ones.

The two original hypotheses were that information technology offers large, sometimes disruptive, opportunities to improve the collection and publication of information relevant to achieving social impact. The second was that an explosion of philanthropy information intermediaries could be wrangled into some kind of collaboration that would lead to a greater-than-the-sum-of-the-parts result [5].

The third hypothesis, forged in the heat of the June 2010 meeting, was that to realize ‘greatest impact’ the social change information infrastructure would need to begin and end with the primary constituents of social change – those who are meant to benefit. It recognized that the collection and signification of information needs to be grounded in the circumstances and experiences of primary constituents if the resulting uses of information are to be transformative. There is a growing set of innovators – like the Family Independence Initiative and LIFT, and the whole civic engagement movement – that are living out this hypothesis. They don’t have all the answers, of course, but they are asking the right questions and making real strides forward. More work like this is needed to figure this out.

Nor are we the first set or organizations to find that we need to discover that we are not cultivating the voices of our primary constituents. Fifty years after the birth of the consumer rights movement, we tend to take it for granted that consumer-facing businesses must listen to their customers. In many ways, we in social change stand where business stood in 1963, before the emergence of a new branch of market research called customer satisfaction. I believe that there is a great deal that we can learn from the experience and craft of customer satisfaction.

How to bring beneficiaries back to the foreground

I have been pondering what it would take to bring the beneficiaries back to the foreground at Markets For Good. The obvious way would be to make sure that the beneficiaries are directly represented. In the anti-apartheid struggle organizations of the 1980s, where I grew up professionally, this was how we did it. The commitment to democratic principles in the way organizations operated meant that meetings – endless meetings it often felt like – and taking decisions took longer. But we learned that there were many unintended benefits to principled inclusiveness, including the time spent learning about each other across the divides of race, gender, class and nationality. The African proverb was often quoted, sometimes tongue in cheek: ‘If you want to go fast, go alone. If you want to go far, go together.’ The national trade union federation, COSATU, spent over a year planning for the first mass national worker stay aways. Their success is often seen as the point of no return in the liberation struggle.

It would be complicated and inconvenient – but not impossible – to figure out ways to represent beneficiaries directly in the Markets For Good enterprise. Maybe now is the time to give this some thought, before it is too late.

Another complementary step that Markets For Good could take occurred to me when I read the paper by GuideStar founder Buzz Schmidt published in February in the online edition of Alliance magazine. Just as the Markets For Good enterprise lacks direct representation from beneficiaries, so does the very understanding of information lack a social dimension. Markets For Good’s working understanding of information seems to be missing the deliberative element that is essential to moving from information to shared meaning, and therefore to greatest impact. Perhaps, I thought, if Markets For Good were to embrace this deliberative element then its priorities and emerging investment strategy would become more people-centred.

Introducing a deliberative element

What does this deliberative element look like? Applied to public reporting, for example, it yields what I have called the Feedback Principle of Public Reporting. This principle holds that when reporting its results an organization should also publish what those who are meant to enjoy those reported results have to say about them [6].

The deliberative element provides a definition of ‘beneficiary voice’ that includes but goes beyond ‘better information services’ to beneficiaries, though I want to acknowledge their utility, both in themselves and as a stepping stone to furthering social change. The deliberative element recognizes that voice is present when constituents find it worth their while to engage with an organization to try to make it better for themselves, their families, and their communities [7]. I have voice when I use my voice and an organization responds to my voice.

We have found that if you ask people a simple question about this you get a clear and accurate answer that tells you a great deal about the relationship between an organization and the people it claims to serve. Applied to the agricultural extension system of the Government of Ethiopia and asked of smallholder farmers, the question is: ‘On a scale of 0 to 10, how likely is it that the Development Agent will listen to what farmers need and try to improve his/her services to you and other farmers?’ Applied to Markets For Good, the constituents are more organizations than individuals, so the question might be: ‘On a scale of 0 to 10, to what extent is it worth your while to engage with Markets For Good to try to make it better for your organization?’ It would be very revealing, I suspect, to see the results of this poll.

When used as a guideline for giving decisions, the deliberative element yields the Constituent Voice element in the new Charity Navigator rating model. That new rating model suggests that donors should ask:

1. Does the charity publish feedback data from its primary constituents?
2. Does the published feedback data include an explanation of how likely it is to be representative of all primary constituents?
3. Does the data include an explanation of why the organization believes the feedback is frank and honest?
4. Is that data presented in a way that shows changes over time going back at least one year?
5. Does the data include questions that speak to the organization’s effectiveness?
6. Does the organization report back to its primary constituents what it heard from them?

Question six directly raises the all-important deliberative dimension. In the customer satisfaction industry they call this ‘closing the loop’ with customers. At least when it comes to information that relates to effectiveness, performance and outcomes, the meaning of information in an ecosystem is best found through deliberation across different constituents of the issue being addressed.
If Markets For Good really envisions a world in which ‘beneficiaries have a voice’, this must mean that funders are presented with high-quality evidence of beneficiary views. Since the beneficiaries are not the payers, their voice counts for more when donors hear it directly. When donors pay attention to their views, then charities will pay attention to them. By requiring charities to answer these six questions, Charity Navigator has taken the largest step ever taken by any development agency that I am aware of to cultivate beneficiary voices [8].

Finally, to come to the vision for the infrastructure contemplated by Markets For Good offered by Buzz Schmidt, the missing deliberative element suggests the following amendments:

The non-profit sector will play an increasingly and recognizably effective role in our social economy and civil society. Its initiatives will continue to capture and offer institutional expression to the hopes, ideas and energies of citizens. But in the near future, supported by strategically coordinated information and transactional (mostly online) services, it will do so in ways that are at once more purposeful, coordinated, deliberative and accountable. Individual donors will seek out and support organizations that are doing work that they value. Institutional donors will be deliberative, accountable, consistent, transparent, intentional and demanding in their philanthropy. Communities will articulate common objectives and track collective progress. Non-profits will report consistently about their own objectives and institutional progress – and what their primary constituents say about those reported objectives and progress. Resources will be directed to organizations that best meet society’s evolving needs. Superior social and environmental progress will result and our liberal democracy be strengthened.

Perhaps by adding this deliberative dimension to the working understanding of information at Markets For Good, the enterprise will find its way to restoring the third hypothesis that was added at the June 2010 meeting: that to realize ‘greatest impact’, the social change information infrastructure would need to begin and end with the primary constituents of social change – those who are meant to benefit. In so doing, it will be far more likely to generate the kind of investments in information infrastructure needed to break out of the prevailing echo chamber of the philanthropy ecosystem. It might also help it grapple with the inconvenient truth that beneficiaries are not currently part of the Markets For Good enterprise, and come to see the exciting opportunity to open the doors to them.

David Bonbright is chief executive of Keystone Accountability. Email david@keystoneaccountability.org

1]  We also noted the irony in referring to the people who must be the primary agents of change as ‘beneficiaries’. The search for a better label goes on. My suggestion is ‘primary constituents’, for we are all constituents of societal problems/solutions, with varying degrees of stake in the outcomes.

2]  The theoretical underpinnings for this are found in the work of Amartya Sen and other contemporary political philosophers; see eg Amartya Sen’s Development as Freedom (Oxford University Press, 1999).

3]  www.marketsforgood.org/about

4]  This parallels a good debate now ongoing in the international development domain about ‘the politics of evidence’ about impact. See eg The Big Push Forward.

5]  ‘As many of you know, the campaign’s goal is to host a conversation about how the social sector uses and shares information, spark new ideas about how to make data sharing and use easier, and highlight and connect efforts already underway.’ www.marketsforgood.org/note-from-the-curator-reflecting-on-2012-looking-ahead-in-2013

6]   For a discussion of the Feedback Principle of Public Reporting in the US non-profit sector context, see Bonbright et al, The 21st Century Potential of Constituency Voice: Opportunities for Reform in the United States Human Services Sector  (Alliance for Children and Families, 2009).

7]  This understanding of voice derives from the work of Albert O Hirschman on decision-making where choices are limited in ways that often pertain for the ultimate beneficiaries of philanthropy. See, especially, Exit, Voice and Loyalty (Harvard University Press, 1970).

8]  It may soon be eclipsed by the World Bank, however, which is looking at how to build Constituent Voice into its government lending for health and human services.

…

They say that the only social problems left to solve are the hardest ones. Poverty, health, global food insecurity, the environment. With tight budgets and insatiable demand for their services, nonprofit organizations are in a tough spot. One solution to make the best of this challenging situation is to make sure that nonprofits are being efficient and effective, and to inform decisions with data.

Luckily, there’s a lot of interest in nonprofit evaluation. Let’s be honest: collecting data about nonprofits isn’t a new idea. But it has moved into some exciting and promising new frontiers in the last few years. For all this progress, though, some field-wide challenges continue to impede nonprofits from systematically collecting and using data.

A few years ago we started a research project to take the evaluation pulse of nonprofits: the State of Evaluation project. In the last round, we surveyed 546 nonprofit organizations from across the U.S. We found that 90% of nonprofits are engaged in some type of evaluation (great!), and we also found several areas with room for growth.

Room for growth. We found that few nonprofits are equipped with the staff know-how, time, and money to meaningfully engage in evaluation.

Time, money, and know-how:When asked about barriers to evaluation, 71% of nonprofits said that limited staff time was a significant challenge to evaluation, 61% said insufficient financial resources were a significant challenge, and 39% said limited staff expertise in evaluation was a significant challenge.

Staffing: Overall, 18% of organizations had a full-time employee dedicated to evaluation. This is an area that’s gaining traction among larger organizations (those with budgets over $1M): in 2012, 53% of large nonprofits had a full-time employee dedicated to evaluation. In comparison, 9% of small organizations (those with budgets less than $500,000) had a full-time evaluation specialist—and these small nonprofits make up three-quarters of the sector. [1- see footnote]

Prioritization: In our research project, we gave the nonprofits a list of ten organizational tasks and asked them to rank these tasks in order of importance. The tasks included: communications, evaluation, financial management, fundraising, governance, human resources, information technology, research, staff development, and strategic planning. These are tasks that most nonprofits are engaged in, to some degree. Evaluation was ranked #9, signalling that evaluation competes with other internal priorities, and usually falls to the bottom of the heap.

These challenges add up to very real concerns. Due to limited time and money, do grant reports contain errors? Are nonprofits or funders basing their decisions on erroneous information? Most importantly, how can we leverage evaluation so that it nonprofits can effectively and efficiently achieve their missions?

How can funders support evaluation capacity? Nonprofits and other social sector organizations help us address our most critical issues. But they can’t do it alone. Nonprofits need to be supported by a strong evaluation infrastructure–like technology, systems, and a general know-how among staff. Based on our research and our experience in the field, we have outlined five areas where funders can play a role to ensure that nonprofits are poised to tackle the world’s biggest social challenges:

Invest in technology, tools, and resources: Online survey software, data analysis software, databases, and equipment (e.g., voice recorders to capture focus group data) can be great assets to grantees.

Support data coaching and training. As the saying goes, “Why worry about Einstein’s pen? Thinking matters most.” Tech tools are most effective when combined with ongoing training about data collection, statistics, research methods, databases, data visualization, and other skill sets.

Support both internal and external evaluation staff. Evaluation serves dual purposes: accountability and learning. External evaluators can bring technical expertise and provide perspectives about accountability. Internal evaluators can provide much-needed follow-up and support for a learning agenda. Internal and external evaluators are increasingly teaming together to support the dual purposes, accountability and learning, simultaneously.

Engage nonprofits in conversations with their peers. Grantmakers are in a unique position to convene peer organizations to discuss issues of evaluation capacity and practice. Peer organizations often have much to learn from each other, and may be able to adopt and build upon each other’s evaluation successes.

Engage nonprofits in conversations with their funders. In State of Evaluation 2012, 75% of nonprofits agreed or strongly agreed that they regularly discuss evaluation findings with funders. When discussing evaluation with funders, 82% of nonprofits agreed or strongly agreed that these conversations were useful. These generally upbeat opinions suggest that the nonprofit community is largely open to evaluation and all that it has to offer.

By investing in evaluation capacity building, grantmakers have an opportunity to lay the groundwork for organizations to collect good data, learn from their work, and improve their results.

[1] Source: Roeger, K. L., Blackwood, A., and Pettijohn, S. L. (2011). The Nonprofit Sector in Brief: Public Charities, Giving, and Volunteering, 2011. Urban Institute, National Center for Charitable Statistics. http://www.urban.org/publications/412434.html

115-200 billion tons moving daily. Not tons of data, but water – in and out of the Bay of Fundy (left) alternating up to 50 ft between low and high tide. Just in case we’re tempted to take the concept of “rising tide” lightly, it’s good to revisit the roots of metaphors with which we may have a bit too much familiarity and appreciate their immenseness. In this case, Michele Jolin, Managing Partner of America Achieves, tracks the rising tide of evidence-based solutions in the social sector and the political alignment happening to drive it on a meaningful scale. Talk about syzygy! (We won’t.That’s pushing the tide thing a bit too far. But,um, we could.)

…

Recently, Results for America joined with The Hamilton Project to host a wide-ranging discussion on the importance of using evidence and data to inform policy and funding decisions. We brought together those who are doing the hard work in communities with those who are in charge of crafting policies at the local and federal levels. At the event, leaders from both parties, and from both ends of Pennsylvania Avenue, called for greater use of evidence in crafting social policies and improving outcomes for young people, families and their communities.

Too often, issues are divided by politics, but during our discussions in April, partisanship gave way to agreement on the importance of working together to confront the challenges facing our country.

Over the next decade, America will face enormous social and economic shifts driven by global competition, an aging population and budget constraints at all levels of government. Young people will be vulnerable to these pressures at a time when we need to be preparing them for lifelong success.

Now more than ever, government funds need to be invested in a way that will produce better outcomes for young people and their families. At Results for America, an initiative of America Achieves, we are committed to directing public resources toward solutions that harness evidence and data to continuously improve and expand the impact of those programs that work in communities across the country.

By collecting data, continuously evaluating programs and directing government funds toward those that are succeeding and away from those that are falling short, we can achieve better results and can get a better return on taxpayer investment. If we can make tax dollars go farther, and have greater impact in communities nationwide, we can begin to restore trust in a government that has proven to be less efficient than the workforce it represents. Something Alan Krueger, the Chair of President Obama’s Council of Economic Advisers, highlighted at our recent event.

President Obama’s Fiscal Year 2014 budget request, released in early April, takes significant steps toward that goal, building on progress he made during his first term. His budget includes an unprecedented focus on improving access to data and increasing the use of evidence in making funding decisions. He expands federal innovation funds and Pay for Success initiatives, increases resources committed to fund evaluations, takes first steps toward using evidence in directing formula grant dollars and begins to create “what works” clearinghouses in federal agencies. The President’s budget also boosts some of the most successful results-based programs created to date to evaluate new ideas and scale up the most effective approaches for improving schools, student achievement and job training, to name just a few. His budget proves that he meant what he said during his 2013 State of the Union address, to “do what works” and “make sure none of our children start the race of life already behind.”

Other opinion leaders and public officials – including New York Times columnist David Brooks, Harvard economist Jeff Liebman, U.S. Senators Rob Portman (R-OH), Mark Warner (D-VA) Mary Landrieu (D-LA) – have touted the demonstrated success and future promise of results-oriented programs. And, thanks to an amendment by Senators Ron Wyden (D-OR) and Debbie Stabenow (D-MI), the Senate Democratic budget calls for greater use of data and evaluation in making funding decisions.

In cities and states across the country, mayors and governors are already adopting an evidence-based approach and using data to improve their communities. The private sector has also stepped up. In order to really improve outcomes, our government cannot be afraid to partner with companies, foundations and communities on what works.

At our recent event, Senator Portman said that government decisions need to be informed by more evidence. He said what many of us know to be true: that in this time of limited budget resources, we have to make sure every dollar is spent as efficiently and effectively as possible, because we cannot afford to spend federal dollars in ways that are ineffective.

The next step will be to build on this growing momentum. Communities large and small are having tremendous success implementing innovative solutions that are improving lives every day. We need to harness their spirit and encourage our lawmakers to drive more public dollars to what works for young people and their families.

I think discussions such as those in the clips here below (excerpted from the recent Global Philanthropy Forum) not only keep our journey human, first and foremost (not getting lost in the technical), but also help us orient ourselves to discover new questions and angles of thought along the way. Many thanks to Sasha Dichter, Chief Innovation Officer, Acumen; Jacob Harold, President and Chief Executive Officer, Guidestar.

Thanks,

Eric

…

Today, President Obama issued an Executive Order — Making Open and Machine Readable the New Default for Government Information

Our take at Markets For Good? It was big news when the President’s FY 2014 Budget was released with a proposal for expanding mandatory e-filing to all tax-exempt organizations. With this Executive Order, it’s great to see the pace continuing en route to fully open data. The importance is not only the accessibility of data, but, rather what we are increasingly being enabled to DO. The sum of all of these new data sets is a living platform for deeper analysis, experimentation and discovery as we attempt to solve tough social problems.

…

More information can be found on these links.

http://www.whitehouse.gov/blog/2013/05/09/landmark-steps-liberate-open-data

http://www.whitehouse.gov/photos-and-video/video/2013/05/09/clock-open-data-executive-order

http://www.whitehouse.gov/sites/default/files/omb/memoranda/2013/m-13-13.pdf

http://www.whitehouse.gov/the-press-office/2013/05/09/executive-order-making-open-and-machine-readable-new-default-government-

Yes. That’s a corn field… but, you could call it a dugout for our purposes today. Daniel Stid, of Bridgespan Group, takes this Field of Dreams cue to ask a few tough questions to you, and to us at Markets For Good. We’re always excited to dig in to real discussion and debate to sort through conversation for ways forward.  There are a lot of ideas buzzing about data that look mighty good on whiteboards and sound great spoken from your wireless mic. And, beyond that (we’re not pessimists), there’s a lot of actual “doing” in the sector as organizations are finding ways to make data work for them and their missions. But will these ideas and work change people’s lives? That’s why we’re here, not “ultimately,” but at the outset. Daniel Stid asks what happens after we build this upgraded information infrastructure to clue us in on how to think – and how to LEAD -  while we work to that end.

…

In its initial concept paper, Markets for Good has put forward an inspiring vision, one of “a social sector powered by information, where capital flows efficiently to the organizations that are having the greatest impact, programs and interventions are more effective and responsive, beneficiaries have a voice, and there is a dynamic culture of continuous learning, development, and innovation.”

Markets for Good proposes that the key to realizing this vision is building out the sector’s information infrastructure, i.e., “the architecture that can help to connect, organize, and structure information so that it can be supplied and used more easily.”

As we undertake this construction, however, let’s ask ourselves whether, if we build it, the putative buyers and sellers in the envisioned marketplace—the philanthropists and nonprofits spending and soliciting money within it—will use it as planned. In short, will better information change their behavior?

A premise of Markets for Good is that we don’t really have such markets yet because the available data is “limited, unstructured, and unused.” But as Friedrich A. Hayek noted in “The Use of Knowledge in Society,” it is precisely in the dim light of these circumstances that markets serve as the best means for allocating scarce resources and in which they flourish. The social sector’s challenge may not be to establish market dynamics, but to temper those already at work.

On Giving and Data

Consider the motives and behaviors of individual philanthropists in this market. Indeed, take out your tax return or checkbook and assess what prompted your own contributions in 2012. Was evidence of impact always the driving concern? Speaking for our household, the other motives included maintaining ties to alma maters, fulfilling religious obligations (and thereby avoiding guilt!), supporting leaders we admire, and giving to causes at the request of friends as they have done in response to our requests. Was your charitable giving prompted by a similar range of motives? How much of it would be influenced by better evidence of impact?

What about the presumably more rational grant-making of institutional philanthropy? I’d like to think that foundations recognize the importance of evidence and thus support promising grantees in establishing the data systems, infrastructure, and evaluations they need to assemble it. I’d also like to think that foundations relentlessly seek out evidence to test their own theories of change and refine them when they encounter countervailing data. In my experience, though, foundations that behave in this way are exceptional relative to a somewhat dismal norm.

Defending Our Positions

The motives and behaviors of nonprofits seeking funding may be just as impervious to better information as those providing it, albeit for different reasons. We shouldn’t be surprised when nonprofits seek to share information about their work in ways that maximize the contributions they receive. Nor should we be surprised when funders’ external demands for evidence of impact leads nonprofits to assume a defensive stance in which they seek to present the best possible case for their performance. Alas, this defensiveness too often prevents organizations from doing the deeper work of measurement, reflection, and learning that could actually improve their results. Will the drive for more transparency pull nonprofits out of their defensive positions, or will it push them to hunker down even more?

In this same light, we could question whether nonprofits seeking funding will readily conform to more standardized definitions of their work and outcomes as envisioned by Markets for Good. More robust markets tend to accelerate the quest for defensible positioning and breed more differentiation, not less—as anyone who has stood bewildered in front of drug store shelves trying to choose among myriad brands, prices, sizes, and features of pain reliever or tooth paste can attest.

The Way Forward

All this is not to say that we shouldn’t improve the social sector’s information infrastructure. As we do so, though, let’s bear in mind that the information produced by it will be used by individuals and institutions driven by a wide range of motives, of which maximizing social impact is but one (and not always preeminent). Individuals and institutions will adapt and use this improved information (or not) to suit their own purposes. If we want more money to go to better solutions and continuous improvement in social impact, we will also continue to need sustained leadership and moral suasion geared toward these ends.

Yumi Sera, Operations Director for the Disability Rights Fund shares the blog post with Mariane Arsenault, an evaluation consultant for the Universalia Management Group, a Canadian consulting firm specializing in monitoring and evaluation, organizational assessment and strategic management for bilateral and multilateral agencies, private sector companies, and NGOs.  In addition to heading up the recent evaluation for the Disability Rights Fund, Mariane has worked on several other evaluation assignments focusing on international development interventions.

…

From the outset, the most important aim was that the M&E System be an extension of our commitment to a rights-based approach and aligned to the principles of the UN Convention on the Rights of Persons with Disabilities (CRPD).

DRF believes that enhancing the participation of persons with disabilities in the realization of their rights is essential to reducing poverty amongst persons with disabilities.

We understand that achieving across the board improved quality of life for persons with disabilities in the developing world is far in the future. DRF’s role is to make grants to change national and local frameworks and attitudes that affect the most marginalized groups in the disability community.

Despite the challenges of M&E in the advocacy realm, we knew that we required a system to collect reliable and valid data about the effects of our grants and to track our collective achievements, as well as to learn from the obstacles we face.

The M&E system we created includes tools and most critically, a logframe with measurable outcome and outputs and SMART indicators. Clear year-to-year milestones guide the work of our Program Officers who provide oversight to grantees through annual Grantee Convenings and frequent contact throughout the life of the projects. Country strategy development and annual assessments provide an in-depth analysis of trends, obstacles, and opportunities for movement building in advancing the rights outlined in the CRPD. Periodic evaluations allow us to reflect on achievements and gaps and change course, as needed.

In late 2012, we tested this system through an independent evaluation conducted by Universalia. This evaluation was one of the first to be commissioned by a grantmaker focusing on supporting disability rights advocacy in the developing world.

A number of challenges for the evaluation were to be expected given the nature of DRF’s work – a grantmaker making modest advocacy grants to disabled persons organizations across numerous developing countries. Data on disability in the countries where we work is scarce, and strong evaluations rely on the availability of quality data. Frameworks for measurement of advocacy achievements are new and yet untested. With DRF’s M&E system and logframe, the evaluation at least had a solid baseline and articulated objectives to measure results.

Traditional evaluation concepts — such as relevance, efficiency and effectiveness – were applied to evaluate hard-to-grasp concepts such as advocacy efforts and movement building. Applying these criteria for DRF’s participatory approach was not impossible, but required adaptation, reflection, and multiple approaches to data collection.

Along with a review of documents, the evaluators conducted focus group discussions with grantees in Uganda and Bangladesh and stakeholder interviews (approximately 90 individuals were consulted). Frameworks based on existing literature were also developed to assess the human rights based approach as well as advocacy and movement building efforts.

The evaluators went into this evaluation with the expectation that they would find very small-scale results, given the fact that many DRF grantees are very small and marginalized grassroots organizations. However, they were surprised by the magnitude of the results achieved.

We have treated the evaluation as an important learning experience – both for the Fund and for the evaluators – and are using the findings for learning conversations among key stakeholders. Here, we share 10 learnings coming out of the experience:

Stick to your principles and values

Being a mission-driven organization, we strive in all that we do to align our work with the principles of the Convention on the Rights of Persons with Disabilities. An evaluation should be aligned with your values.

Seek to empower the people you are working with

We believe that meaningful change will happen only when persons with disabilities are empowered to take decisions that affect their own lives. Thus, it is crucial to involve and listen to the people your intervention seeks to benefit.

Commit your organization to robust monitoring and evaluation

Our governing body, global advisors, and staff were involved in and committed to measuring the results of our work and learning from the process. Creating an organizational culture of learning can help you and your evaluators tread into new territory.

Be accountable to your donors and to the public

The evaluation helped us to report our progress to our donors and to the public. To be accountable for public funds, you should have an independent entity validate assumptions and interventions and point out new areas to explore.

Engage stakeholders — learn from them and educate them

We network with stakeholders in global fora and meetings. Collaborating with others creates a rich environment for movement building that is essential to work for social change. Sharing lessons helps build a larger and stronger movement.

Adapt and refine monitoring and evaluation tools

To some, traditional data collection and measurement may seem too linear or quantitative for the complexity of the social sector, but by articulating a theory of change and logframe, we have strengthened our tactics. Combining quantitative measurements with stories can be a powerful portrayal of your mission.

Be rigorous in your analysis

Collecting data about each grant seems like a micro-managed process, but aggregating and analyzing data illustrate how we are (or are not) having an impact. Developing a shared meaning of concepts and terms across your organization can help you to standardize your monitoring and reporting, especially across diverse populations and geographic areas.

Review and evaluate processes, in addition to results

One of our biggest learnings for our evaluation was that the process, especially in advocacy work, is as important as the intended results. How you succeed or fail can be more of a learning than the attainment of a goal. A theory of change helps articulate and make these processes visible.

Apply the learning to your work

Whether or not we agree with the findings, we know that the evaluation has contributed to a healthy dialogue in our organization about our work. You can use the opportunity of an evaluation to test and question your assumptions and become a better organization.

Contribute to the field of social change

Sharing what we are doing with a wider audience can raise awareness of the rights of persons with disabilities. We also hope that our approach to monitoring and evaluation will contribute to the exchange of ideas about evaluation in the social sector and among grantmakers.

See http://www.disabilityrightsfund.org/evaluation for our evaluation and our recent report on our first years, “One in Seven” http://www.disabilityrightsfund.org/oneinseven.

…