Spotlight on the Challenge of Measuring Social Impact

By Samuel Casanova (originally published on SimpleChange)

What does it mean to be successful?

A successful social venture 1) generates an impact, like causing a positive change in its target population, and 2) is economically sustainable, i.e. it “can pay the bills” and has the capacity for growth reinvestment.

Both issues are headaches for entrepreneurs, but measuring social impact is even more of a riddle.

Company improve its social impact (work on influence marketing).

What do we mean by impact?

To understand social impact, it’s useful to be able to answer the question of Dan Levy, a professor at Harvard Kennedy School:

What happened (with the program/service/product) – and what would have happened without it (counterfactual) = IMPACT

Of course, there are different types of scenarios regarding impact: positive impact, negative impact, positive-negative impact, and null impact – in the event that a person using your product or service stays the same as a person who does not.

When do we measure the impact?

“The issue is not the impact assessment, it is the clarity of your objectives,” said Ana Paula Hurtado, founder of Something In Common A.C., a social organization that evaluates social projects of nonprofits, entrepreneurs, business, and government.

And though it may sound harsh, she recommends that the expectations we have of impact be lowered, because if there is no certainty of purpose, it cannot be measured and applied well.

The impact does not begin once the program/product/service is up and running. It starts practically from the formulation of the concept. It must have a well-defined mission and theory of change that it is intended to promote.

Mission Target Aspirations Motivation Goals Concept

Regarding the mission, the more concrete and more specific it is, the better. Focus on what it is that you want to do, who is intended to benefit, and what is expected.

Kevin Starr, director of Mulago Foundation, says that a good mission should be able to be built with only eight words (or less) and must cover three aspects:

  • Verb
  • Target population
  • Measurable result

For example:

  • Families build homes with families.
  • Preventing HIV infection in Brazil.
  • Connecting people through lending to alleviate poverty.

Change theory answers the question: How, in theory, does our program/product/service solve the problem? It involves a process of logical thinking to reach the ideal state.

“One of the big mistakes of entrepreneurs is they have no theory of change,” says Ana Paula Hurtado. “They do not measure change in behavior. They measure intervention.”

However, there are many ways to explain the theory of change. Ana Paula Hurtado explains that you should consider:

  • Supply
  • Products
  • Results

Inputs are what you need (activities, materials, etc …) to make your product. The product is the effect of all inputs “joining together” perfectly, and the result is how it improves, grows, or evolves your target population after using your product /program /service.

Consider, for example, online English classes. The inputs could be the platform where we give the classes, the curriculum and teachers; the product (intervention) are classes, and this is precisely what entrepreneurs tend to measure as the product. And this is the error.

The result would be that participants learn English, but here is not just the theory of change: How do you measure it?

You need an indicator that is specific, measurable, attainable, and realistic. An indicator that you can continue measuring in 2, 3, 5, and 10 years and that is inspirational (based on the methodology of Something In Common A.C.).

In the case of English classes, they would run the TOEFL or IELTS tests as an indicator.

You define yourself based on your goal, or you can find the appropriate indicators.

The challenge of impact assessments

“Not everyone is willing to pay for a rigorous impact assessment,” says Ana Paula Hurtado. This reflects one of the great challenges of entrepreneurs regarding impact assessments: lack of resources. As a result, you tend to want to make impact assessments that are actually evaluations of results or processes.

An impact assessment will not help if there is no clarity on your goals.