How do you know when what you’re doing is working? In school, it’s easy to measure success; you take a test and you’re assigned a grade. In farming you’re rewarded with crops. In business, it’s sales and net income. But when trying to change the culture or make a social impact, it becomes more difficult to calculate an ROI.
I’ve had a few conversations with stakeholders over the last couple of weeks about what success really looks like when trying to evaluate the social benefit of our rickshaw sales. There seems to be a spectrum of opinion about what impact they want to see us make and how they want to know we’re making it: Some are content with number of rickshaws sold since we established in the business plan there’s a need. Others are interested in seeing the difference in people’s lives from meeting them or reading client stories. Still others want to quantify it in some way.
For me, each of these preferences is fine. After all, they are preferences, a selection: something chosen, favored, or held above other things in estimation. But I think we just need to be honest about what each of these pieces of information tells us and accept the information for what they are—nothing more, nothing less. Each of these ways of measuring impact is different:
Number of rickshaws sold can mean several things. It means that someone has access to a source of income from each of those vehicles. It doesn’t mean they use it all of the time or that it’s the only source of income. Some of our clients buy a rickshaw and then buy a trolley so they can carry goods at night after the demand for conveyance has gone down. This gives one person double income. Additionally we don’t know if just one rickshaw driver is using the vehicle or if he lends it to a family member.
Anecdotal data tells a different story. Our client Ajay has finished his installments and is the owner of his rickshaw. After joining SMV, he was able to accumulate enough money to find better housing and send his two daughters to school. He has well-made clothes and a sense of confidence about him. He is an example of how someone can use the rickshaw asset as a step up the socioeconomic ladder. Hearing about his life and the improvements for him and his family makes us feel good about the work we do. It demonstrates the potential path someone can take as a customer. But Ajay’s story isn’t representative of all clients. Some use the income for more paan and alcohol. Some cannot make the installment payments and see their rickshaw repossessed. It’s an “n” of one.
Quantifying our impact can be difficult to tease out, and these numbers can communicate a variety of things. One of the more common ways to measure impact in numbers is by using income as a proxy for livelihood improvement. We can calculate our clients’ estimated increased income from asset ownership instead of the daily rental alternative: how much people are saving from not having to pay the rental fee. However, this figure is an estimate—it’s the likely result of rickshaw ownership with us. It assumes that the rickshaw driver rented every day. It assumes that our client rickshaw driver will only work eight hours (the maximum time fleet owners usually offer rickshaws on rental.) We also don’t know how this money is being spent. All of the income could be going to family needs; it could be remitted back to other family; it could be paying back other debts; it could be going straight to the paanwalla or gambled away in cards.
While all of these numbers are useful and tell us different things about the business and its effect on people’s lives, as a good student of performance management, I would prefer to measure outcomes, or what happens as a result of outputs. Unfortunately this is often cost prohibitive.
To unpack this idea a bit, I’ll define these terms vis-à-vis our business. Inputs are working capital or the rickshaws that the working capital buys. Outputs are rickshaw owners and vocational support we provide them. Outcomes are lives improved as a result of rickshaw ownerships—increase in nutritional intake, improved housing, better healthcare provision, or better quality education. Outcomes tell us the end result of whatever intervention is put in place.
I’m really interested to know what our clients are able to do as a result of joining SMV. Do they feel more secure having municipal licenses and insurance? Are they happier and more confident being an owner instead of a renter? Have they been able to provide for their families more?
On the other hand, connecting a relationship with SMV to a client’s livelihood improvement is dangerous. Conditions external to owning a rickshaw could result in livelihood improvement whether they are a government health insurance scheme, family remittances, or household additional income. Likewise, there are conditions that could result in detrimental effects like deaths in the family, extreme weather, disability, etc. SMV cannot take credit for these positive or negative changes in quality of life as there are other factors at play, and isolating the repercussions of rickshaw ownership (or any other development intervention) is a complicated task plaguing researchers.
So why, you ask, should we measure impact at all if it’s so difficult to pinpoint? The answer is simple. Because these metrics do tell us something. And because we should want to know the answer. So many development projects and well-intentioned initiatives don’t even start to make a dent in the underlying cause they are trying to improve. And so many have unintended consequences. I believe that asking the question, “What impact are we having?” is an important one. It helps those trying to make a difference keep the end goal in mind and make sure that the activities they are doing are in alignment with that goal.
We at SMV are trying to bring dignity to the profession of rickshaw driving and help these drivers improve their livelihoods through asset ownership and vocational support. The day we stop asking what impact we’re having is the day we stop caring about our mission and start just selling rickshaws. While we are, indeed, a business, the day we just sell rickshaws is the day we stop being who we are.
Going forward I hope to find a good proxy for livelihood improvement that wouldn’t necessitate a full-scale, statistically significant survey (no matter how much fun I would have washing those variables through a few regressions.) I’m hoping that our academic consultants can find a correlation between one of the indicators and general livelihood improvement, so that we could measure just one or a few indicators to decrease our reporting burden.
In addition to improved livehoods, SMV would like to see an increase in the collective self confidence of rickshaw drivers and the occupation become more organised. But how will we know? These are long term goals, and it will be tricky to measure their success. However, there are a few indicators we can use to evaluate this impact. We can look for prices to normalise with minimum fares, standardised apparel, decreased facilitation payments for licenses, fewer reported incidences of police harassment, to name a few. Certainly we will look for these indicators as time progresses, but we will also know that they are only indicators and that we cannot prove causation with correlation.
All of these ways of evaluating impact are useful. They just tell us different information. My biggest concern is that we acknowledge what information we’re using and recognize its limitations. We can’t assume that increased income is used for livelihood improvement. We can’t assume that one good client story or one bad client story means that many other clients are having the same experience. We can’t assume that the number of rickshaws sold means that the same number of families are enjoying increased income. Nor can we assume that improved client living conditions are completely the result of joining SMV.
What we can do is be good consumers of information and realise the limitations of what these indicators and numbers communicate. We should try to measure (and maximise!) impact while being realistic about what we can learn from the data. We will be careful determining what impact we’re having on the rickshaw industry, and I urge all of us to be careful consumers of so-called “impact” metrics. But let’s keep measuring lest we lose connection with what we aim to do and why.