Lessons Learned: Common “Metrics for Success” are not successful metrics at all

Building on the last “Lesson Learned” I wrote about with regards to partnering to make changes, this article is about is another lesson we are continuing to learn: how can we rate the success of the partnerships and programs we implement?
One of the dangerous myths about NGOs is that their success can be rated on their “Program to Overhead Ratio”.  Not only is this wrong, as in this ratio does not help distinguish good from bad NGOs, but relying on this ratio is actually harmful to the NGO world.  Why?  Many reasons:
1) This number can be manipulated in so many ways, it is shocking to those who consider those numbers as representatives of static facts.  The first time we filed our taxes at PEPY and I was asked by accountants “how do you want to split up your office overhead?” I was confused.  “Isn’t office overhead, ‘overhead’?”  Nope, turns out it’s not always.  Actually, since our office is in Cambodia and the reason to have the office there is to operate programs in Cambodia, we could legally claim ALL expenses incurred in Cambodia as program costs.  ALL!  You can take your rent and divide it by floor space, by hours used, by any systematic way of saying what part of your rent (and hence then internet, water, etc) goes to which program.  And, like I said, legally ALL of our in-country operations could be considered programs.  We do not file our taxes that way, but for many big organizations with offices in Cambodia, New York, etc, the ENTIRE expenses of the offices overseas in the areas where the work is being done are written off as program expenses.  100%.  So if you look at the financials of one of those organizations and say, “Wow, well this is a ‘good’ organization, they only spend 15% on overhead” consider the fact that that 15% is likely the costs of just those things they consider “overhead” within their foreign offices and all of the offices, including things like thousands of dollars per floor for “decoration expenses” (not joking, these exist in many INGOs), etc are all incorporated into their program expenses.  That 15% no longer looks so “lean”, does it?  By deciding to donate strictly based on program to overhead ratios, this is incentivizing some groups to misrepresent their actual spending and can lead to investing in programs which are not so effective at all.
2) Not “following the plan” does not mean the NGO did something wrong… more often than not they did something right!  If you haven’t already, go out and read the first 15 pages (at least) of William Easterly’s book, “White Man’s Burden” to understand why judging an NGO by their ability to successfully follow through on their pre-determined plan is not only wrong but harmful to our planet and its people.  I am glad that I read his ideas after I had already been working in development for a few years and had already made my own perceptions with regards to how the majority of the NGO world functions.  When I did finally get around to reading through the initial chapters of his book all I could do was smile, agree whole-heartedly, and want to make his book required reading for ANYONE planning to donate even the smallest amount of money to an NGO.  He had put into words what I was seeing and living every day – the NGO world is DESIGNED to promote inefficiencies.  How do I mean?  Let’s look at large grant making.  Here is an example process of how a large multi-country grant might be funded/executed:

1. Someone comes up with an idea for change (often times a government official or an academic and very often someone far removed from the areas meant to be served).
2. A larger body, perhaps a UN agency, comes up with a plan and “log frame” – or a set of actions and metrics for success which the recipient organizations are meant to follow.
3. Layers of NGOs apply for a grant and are required to set their log frame and measure performance metrics before they have even started the project, when knowing these numbers and this process is unreasonable.  Even if they DO know already based on past experience that the process or metrics is flawed, it is unlikely as you get down the chain that the NGO would be able to alter the defining grant structure at this point so they have to choose to accept the grant or not based on its entirety, thereby forcing NGOs to either turn down money or accept that they will be inefficiently using funds before they even start.  (I was just at a meeting of education NGOs this morning where we were discussing how we might collectively apply to a grant which defines success in a way that most of us agree is not completely in line with how we currently define it. Do we comprise, or look elsewhere for funding?)
4. The funding filters through a few layers of NGOs, perhaps down to a grassroots group enacting the project (with of course significant funding going to cover administrative costs to the filtering NGOs all along the process).
5. The NGO finally starts working. Just after starting, they might realize that the assumptions from which they have been operating were wrong, i.e. they need to dig different kinds of wells because the ones budgeted for break too easily or the education program they started is too complicated and they will need a more community-based approach to get people involved, etc.

The problem is, they now are not able to change the way their project is run, or not easily at least, and if they do try to change it or do not complete the task as originally outlined, it is likely that they will not receive all of the funding and will surely not be eligible to receive more funding next year.  Hence, NGOs that apply for grants in these types of systems are currently incentivized to continue on with projects even if they have identified the project is flawed, could be done better, and in some cases, is causing harm.
3) Investing because of the WRONG metrics does nothing to incentivize NGOs to focus on the RIGHT measures of success.  What are the right ones (in my opinion)?  Things that further them along on their mission.  If NGOs are incentivized to take actions which further their mission, then investment decisions in NGO work can focus on choosing the mission that fits with your own view of a better future as well as measuring an NGO’s success at working towards those goals.  For example, an NGO whose mission is increasing literacy in an area might monitor these metrics for success: changes in literacy rates, enrollment in literacy focused classes/schools, density and use of libraries, survey public opinion about the reading and its value in society, etc.
(Note: a whole different lesson we have learned is on setting the RIGHT metrics for success.  Often these are not the measures that large organizations report.  For example, would an NGO whose mission is to bring quality healthcare into a rural area be best judged on: a) the number of clinics they built in a given time or b) the frequency that those clincs are used, hours the doctors are present, number of patients cured, etc?  More often funders focus on the former, for lack of ability to monitor large projects and because quantitative data is often seen as more valid than qualitative experiences.  Unfortunately, from what we’ve seen on the ground, whether it’s a clinic, library, school, or training center, buildings can and often do sit vacant.   In my opinion, one successfully operating health clinic is way more worthy of investment than 100 non-functioning ones, but I digress…)
4) If we are judging NGOs based on their ability to work towards fulfilling their mission, how do we determine what metrics we should use which are in line with that mission?  We should let NGOs set their OWN metrics for success, I believe.  Why?  Because we just won’t get it.  THEY are there and they know more about the barriers and needs they are dealing with in a certain area.
For example, with PEPY’s Bike-to-School Program, some might think “success” would be measured by the number of kids who ride their bike to junior high school vs. their parents using the bike or selling it.  I would define success as how many parents allow their kids to stay in school LONGER to get the bike in the first place, no matter what is done with it afterwards.  In the first part, you would be focusing on junior high school records, in the second, primary school records.  PEPY’s Bike-to-School Program might be deemed “successful” by looking at  6th grade graduation rates and “unsuccessful” by focusing on junior high school attendance, for example.  (Though in the case of the real Bike-to-School Program, I think we would rate it success on both accounts, but I digress again….)
Granted, it is within our rights, or I would argue, it is our obligation, to ask for clarification from NGOs on the hows/whys of their decisions (see “Is PEPY making a good decision?” for an example of our donors and supporters doing the same for us).  If we don’t agree with the logic in the NGO’s decision-making or don’t think the metrics for success are indeed measuring progress towards their mission, than it is our obligation to once again take action: a) learn more and feel comfortable with the clarification, b) make suggestions and requests for change, and c) invest only in the programs we believe in, and remove our funding from the ones we don’t.
We get to vote with our money.  Which do YOU think is a good NGO?  Go out and support those ones and cast YOUR votes.  By aligning our investments in NGOs with our own vision for a brighter future and our own thoughts on measuring success, we can move towards the changes we want to see in the world.  By recognizing that common metrics for determining an NGO’s success can often point us in the wrong direction, we need to first educate ourselves and the general donating public about these misconceptions and then go out and ask more of our NGO sector, start a dialogue with them about how they define success, and cast our votes accordingly.  Then, when all of the funding starts flowing to the NGOs that are transparent about their decisions, admitting their mistakes, and aligning their actions with their missions, we can change what it means to be a “good” NGO and start investing in the projects which are having the biggest and best impacts.