{"id":4652,"date":"2012-11-21T09:59:55","date_gmt":"2012-11-21T02:59:55","guid":{"rendered":"http:\/\/pepyempoweringyouth.org\/?p=4652"},"modified":"2012-11-21T09:59:55","modified_gmt":"2012-11-21T02:59:55","slug":"lessons-learned-our-external-evaluation","status":"publish","type":"post","link":"https:\/\/www.pepyempoweringyouth.org\/lessons-learned-our-external-evaluation\/","title":{"rendered":"Lessons learned &#8211; our external evaluation"},"content":{"rendered":"<p><a href=\"https:\/\/pepyempoweringyouth.org\/lessons-learned-our-external-evaluation\/lessons-learned3\/\" rel=\"attachment wp-att-4713\"><img fetchpriority=\"high\" decoding=\"async\" class=\"alignleft  wp-image-4713\" title=\"lessons-learned3\" src=\"https:\/\/pepyempoweringyouth.org\/wp-content\/uploads\/2012\/11\/lessons-learned3.jpg\" alt=\"\" width=\"350\" height=\"263\" srcset=\"https:\/\/www.pepyempoweringyouth.org\/wp-content\/uploads\/2012\/11\/lessons-learned3.jpg 500w, https:\/\/www.pepyempoweringyouth.org\/wp-content\/uploads\/2012\/11\/lessons-learned3-300x225.jpg 300w, https:\/\/www.pepyempoweringyouth.org\/wp-content\/uploads\/2012\/11\/lessons-learned3-195x146.jpg 195w, https:\/\/www.pepyempoweringyouth.org\/wp-content\/uploads\/2012\/11\/lessons-learned3-50x38.jpg 50w, https:\/\/www.pepyempoweringyouth.org\/wp-content\/uploads\/2012\/11\/lessons-learned3-100x75.jpg 100w\" sizes=\"(max-width: 350px) 100vw, 350px\" \/><\/a>Earlier this year, PEPY hired an external consultant to conduct an impact assessment of our programs. You can read a summary of the final report and also the response to the report from the PEPY Board and Senior Management team via the links below.<br \/>\n<a href=\"https:\/\/pepyempoweringyouth.org\/wp-content\/uploads\/2012\/11\/Executive-Summary-of-Program-Evaluation-2012.pdf\" rel=\"attachment wp-att-4653 noopener noreferrer\" target=\"_blank\">Executive Summary of Program Evaluation 2012<\/a><br \/>\n<a href=\"https:\/\/pepyempoweringyouth.org\/wp-content\/uploads\/2012\/11\/Board-and-Senior-Management-response.pdf\" rel=\"attachment wp-att-4654 noopener noreferrer\" target=\"_blank\">Board and Senior Management response<\/a><br \/>\nWhile the impact assessment did provide some insight into our programs, our biggest learning from the experience was with regards to the value of conducting an external evaluation. The advantages of hiring an outside evaluator are that the assessment can be more objective and provide insights that team members might not be able to contribute themselves &#8211; therefore as a whole providing a more credible analysis.<br \/>\nOur experience, however, was that these advantages are not necessarily guaranteed, and were also far outweighed by other disadvantages \u2013 some of which we had understood in advance and accepted, and some of which we had not anticipated.<\/p>\n<ul>\n<li>An external evaluator is not always able to understand an organization\u2019s beneficiaries and programs as well as the organization itself. Unlike some organizations, a large proportion of our staff not only spend large amounts of time in the communities they work with, but also are actually from those communities themselves. Therefore many of the findings and recommendations of the assessment did not really tell us anything new.<\/li>\n<li>At one point we had to encourage the evaluator to be \u201cmore critical\u201d and just \u201ctell us the bad bits\u201d \u2013 we didn\u2019t want a glowing report to give to our donors, we were keen for insight into what we could do better. In retrospect we would probably have been more critical of ourselves.<\/li>\n<li>In our opinion, the evaluator \u00a0had limited time at our projects sites, which restricted his awareness of the scope of our work, and therefore his evaluation, to be in a position to write a report about certain projects. Methods of gathering information focused heavily on interviews with our staff and with beneficiaries that we interact with everyday, as well as gathering information from our published reports. Again, much of this information we therefore knew already and did not provide new insights.<\/li>\n<li>External evaluators are expensive \u2013 while we are committed to understanding our impact and communicating this to our donors, in retrospect we are not convinced that this method was the best way to spend that amount of money.<\/li>\n<li>We do not have reliable baseline data for many of our projects so there was a limit to the amount of quantitative evaluation that could be conducted. This we were obviously aware of in advance but it therefore meant a need for rigorous qualitative evaluation which we felt was not conducted strenuously enough.<\/li>\n<\/ul>\n<p>So what is our take-away learning from this? Well, there are two main things:<\/p>\n<ul>\n<li>If we hire an external evaluator again, we need to spend more time and energy in briefing this person. We would need to do a better job at ensuring a thorough understanding of PEPY, making our expectations clearer and being firmer about our expectations about how the evaluation would be conducted.<\/li>\n<li>Most likely, though, we probably wouldn\u2019t hire an external evaluator in the same way again! We would be more likely to hire, for lower cost, a consultant from a respected partner organization who is familiar with our work and has a better understanding of what kind of organization we are and what we are hoping to gain from an evaluation. While some could argue that this may be a less objective evaluation, we believe it would actually be more critical and more useful to us, and be a more efficient way of spending our funds.<\/li>\n<\/ul>\n<p>Karina Kloos, a member of our board Programs Committee also noted: \u201cPEPY\u2019s experience in this case seems to highlight some of the problems with evaluation pressures from donors across the industry. NGOs can often feel obliged to accept the idea that more \u201cprofessional\u201d assessments are necessary and legitimate the evaluation process and results. This is not only often not true, but also can devalue local staff in their ability to critically examine their own projects, report mistakes and share lessons learned. Relying on external evaluators is not always a way to ensure that organizations are being realistic and honest. It\u2019s also worth noting that placing demands on NGOs to invest their limited resources in evaluation fuels the \u201cCatch 22\u201d of NGO assessment which criticizes organizations for spending too much money on overheads\u201d.<br \/>\nFor those interested in reading more about the pros and cons of external evaluations for NGOs, you might be interested in <a href=\"https:\/\/www.bond.org.uk\/data\/files\/Effectiveness_Programme\/Briefing_on_practices_and_debates_in_evaluation.pdf\">this article<\/a>. Produced by Christina Laybourn, formerly of\u00a0Bond, a UK membership body for NGOs working in international development, it gives a good overview of the pros and cons of differing approaches used in the development industry.<br \/>\nWe\u2019d love to hear your thoughts on this or any related experiences you may have to share!<br \/>\n&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Earlier this year, PEPY hired an external consultant to conduct an impact assessment of our programs. You can read a summary of the final report and<span class=\"excerpt-hellip\"> [\u2026]<\/span><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[55],"tags":[],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.pepyempoweringyouth.org\/wp-json\/wp\/v2\/posts\/4652"}],"collection":[{"href":"https:\/\/www.pepyempoweringyouth.org\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.pepyempoweringyouth.org\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.pepyempoweringyouth.org\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.pepyempoweringyouth.org\/wp-json\/wp\/v2\/comments?post=4652"}],"version-history":[{"count":0,"href":"https:\/\/www.pepyempoweringyouth.org\/wp-json\/wp\/v2\/posts\/4652\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.pepyempoweringyouth.org\/wp-json\/wp\/v2\/media?parent=4652"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.pepyempoweringyouth.org\/wp-json\/wp\/v2\/categories?post=4652"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.pepyempoweringyouth.org\/wp-json\/wp\/v2\/tags?post=4652"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}