- Building donor agency and partner countries’ capacity on effective monitoring and evaluation are key components of aid effectiveness. To learn how to undertake effective evaluations follow ADB/IED series Evaluation Guru.
- By strengthening capacities to effectively evaluate development projects, the groundwork for accountability and learning is laid. To learn how to conduct program evaluations, watch the latest episode of Evaluation Guru.
Evaluation Guru is an evaluation capacity development video series launched by Asian Development Bank’s (ADB) Independent Evaluation Department (IED). As part of IED’s Evaluation Academy initiative, the series takes the multimedia approach to meet the growing demand for evaluation capacity development in ADB’s client countries. The series illustrates how to design, implement, and manage evaluations which in turn will guide programs and projects towards development effectiveness, foster accountability, and learning. In the fourth episode of Evaluation Guru, former IED Director General Marvin Taylor-Dormond talks about Program Evaluation.
Transcript
Evaluation Guru Ep 4
Program Evaluation
Hello and welcome to Evaluation Guru. Through this series we will share with you how to design, implement, and manage evaluations which in turn will help you improve the development effectiveness of your projects, programs, and strategies, as well as foster accountability and learning.
Marvin Taylor-Dormond
Former Director General
Independent Evaluation Department
Asian Development Bank
We’re talking in this episode about program evaluation. When we speak about program evaluation we refer to programs, projects, strategies, policies. To conduct program evaluation we will use the same seven steps that we talked about in a previous episode. And so the first step to conduct a program evaluation is to identify and define the object of evaluation. In this case, clearly understand the program, the project, the strategy, or the policy that you are going to evaluate. This is the foundation of your program evaluation. You must understand the objectives, the context, the budget, the resources, the mechanisms of impact intended through the program that you’re about to evaluate. It is a good practice to complete this first step with the definition of what we call a Theory of Change.
The second step is to define the evaluation criteria. For sovereign projects in Multilateral Development Banks, we use criteria such as relevance, effectiveness, efficiency, sustainability, coherence, and impact. For higher plane evaluation the criteria or the areas of evaluation depend on the strategy or the policy that we are assessing. It is important that you clearly define what areas of the program you want to assess. And clearly focus your evaluation on an overarching question. The subordinating questions should support the overarching question. For this, it is a good practice to use the principle of Mutually Exclusive and Collectively Exhaustive set of questions.
The third step, define the standards. There are certain standards that allow us to determine whether the program or project or strategy is highly relevant, or just relevant, or below par. For more complex evaluations, what we call higher plane evaluations, you should understand the different dimensions of your program to come up with an approach paper for your evaluation. You can consult the set of approach papers that have been produced in Independent Evaluation Department (IED) by visiting the IED’s site.
The fourth step is to define the methods that you’re going to use to measure the program according to each of the criteria. And you may use experimental methods, surveys, interviews, document reviews, statistical analysis, beneficiary analysis, contribution analysis. It is important that you set this clearly in what we call an evaluation design matrix.
The fifth step is to measure the degree of accomplishment of your object with the criteria, and to compare this with the standards. Proceed with this measurement using the method that you defined in the previous step. And always be ready to adjust your approach once you are measuring the degree of accomplishment. You may find surprise in the field by reading documents or by applying any of your statistical analysis.
The sixth step is to put the evidence together and come up with your overall assessment. Sum up the assessment by criteria and to establish the overall assessment of your program by summarizing this criteria.
You have all the pieces together now and you come to the seventh and last step, writing your evaluation report. It is important that you keep in mind some fundamental principles to write your program evaluation. The first place, use your theory of change, the one that was defined in step one, to describe what worked, what didn’t work, and why? What where the channels of impact? Did they work well? Was it well defined? Were there failures in understanding the circumstances on the ground? Secondly, answer the questions that you established in step 2. The overarching question has to be clearly answered and the subordinating questions as well. Third, identify key lessons that are derived from your assessment. Fourth, identify your key issues that the program has faced during implementation. And finally, fifth, write your recommendation. Keep the recommendations very basic, addressing only the fundamental issues you have identified in your evaluation. And always remember, you evaluate with a retrospective approach but always with a prospective intention of improving operations in the future. You may always use the lessons learned, the issues you identified, the recommendations to improve other similar programs in the future.
Use is a concept that must permeate the mind of the evaluator from the beginning, from inception. You need to spend time with the evaluee, with the stakeholders. Listen to them, develop trust, so that the evaluees and the stakeholders in general are able to use your evaluation report in a productive way. We will devote more time to “use” in an upcoming episode, and for now thank you very much.
Thank you for watching Evaluation Guru. We will be back with another episode on how to design, implement and manage effective evaluations. Till then, don’t forget to look up our social media accounts. Bye for now.
