Lessons learned – our external evaluation

Earlier this year, PEPY hired an external consultant to conduct an impact assessment of our programs. You can read a summary of the final report and also the response to the report from the PEPY Board and Senior Management team via the links below.
Executive Summary of Program Evaluation 2012
Board and Senior Management response
While the impact assessment did provide some insight into our programs, our biggest learning from the experience was with regards to the value of conducting an external evaluation. The advantages of hiring an outside evaluator are that the assessment can be more objective and provide insights that team members might not be able to contribute themselves – therefore as a whole providing a more credible analysis.
Our experience, however, was that these advantages are not necessarily guaranteed, and were also far outweighed by other disadvantages – some of which we had understood in advance and accepted, and some of which we had not anticipated.

• An external evaluator is not always able to understand an organization’s beneficiaries and programs as well as the organization itself. Unlike some organizations, a large proportion of our staff not only spend large amounts of time in the communities they work with, but also are actually from those communities themselves. Therefore many of the findings and recommendations of the assessment did not really tell us anything new.
• At one point we had to encourage the evaluator to be “more critical” and just “tell us the bad bits” – we didn’t want a glowing report to give to our donors, we were keen for insight into what we could do better. In retrospect we would probably have been more critical of ourselves.
• In our opinion, the evaluator  had limited time at our projects sites, which restricted his awareness of the scope of our work, and therefore his evaluation, to be in a position to write a report about certain projects. Methods of gathering information focused heavily on interviews with our staff and with beneficiaries that we interact with everyday, as well as gathering information from our published reports. Again, much of this information we therefore knew already and did not provide new insights.
• External evaluators are expensive – while we are committed to understanding our impact and communicating this to our donors, in retrospect we are not convinced that this method was the best way to spend that amount of money.
• We do not have reliable baseline data for many of our projects so there was a limit to the amount of quantitative evaluation that could be conducted. This we were obviously aware of in advance but it therefore meant a need for rigorous qualitative evaluation which we felt was not conducted strenuously enough.

So what is our take-away learning from this? Well, there are two main things:

• If we hire an external evaluator again, we need to spend more time and energy in briefing this person. We would need to do a better job at ensuring a thorough understanding of PEPY, making our expectations clearer and being firmer about our expectations about how the evaluation would be conducted.
• Most likely, though, we probably wouldn’t hire an external evaluator in the same way again! We would be more likely to hire, for lower cost, a consultant from a respected partner organization who is familiar with our work and has a better understanding of what kind of organization we are and what we are hoping to gain from an evaluation. While some could argue that this may be a less objective evaluation, we believe it would actually be more critical and more useful to us, and be a more efficient way of spending our funds.

Karina Kloos, a member of our board Programs Committee also noted: “PEPY’s experience in this case seems to highlight some of the problems with evaluation pressures from donors across the industry. NGOs can often feel obliged to accept the idea that more “professional” assessments are necessary and legitimate the evaluation process and results. This is not only often not true, but also can devalue local staff in their ability to critically examine their own projects, report mistakes and share lessons learned. Relying on external evaluators is not always a way to ensure that organizations are being realistic and honest. It’s also worth noting that placing demands on NGOs to invest their limited resources in evaluation fuels the “Catch 22” of NGO assessment which criticizes organizations for spending too much money on overheads”.
For those interested in reading more about the pros and cons of external evaluations for NGOs, you might be interested in this article. Produced by Christina Laybourn, formerly of Bond, a UK membership body for NGOs working in international development, it gives a good overview of the pros and cons of differing approaches used in the development industry.
We’d love to hear your thoughts on this or any related experiences you may have to share!