- Evaluation, as Michael Scriven said, is a process to determine the worth, the merit, the significance of the object of evaluation. Evaluations are the products of this process.
- What does Independent Evaluation mean. Independence in this context means to be free, to be protected from material threats to objectivity.
- Independent Evaluation is the process of determining the value, the merit, the significance of an object of assessment in an objective manner.
Evaluation Guru is an evaluation capacity development video series launched by Asian Development Bank’s (ADB) Independent Evaluation Department (IED). As part of IED’s Evaluation Academy initiative, the series will take the multimedia approach to meeting the growing demand for evaluation capacity development in ADB’s client countries. The series illustrate, how to design, implement and manage evaluations which in turn will guide programs and projects towards development effectiveness, foster accountability, and learning. In the third episode of Evaluation Guru, IED Director General Marvin Taylor-Dormond talks about the concept of Independent Evaluation.
Transcript
Evaluation Guru Ep 3
What is Independent Evaluation?
Hello and welcome to Evaluation Guru. Through these series we will share with you how to design, implement, and manage evaluations which in turn will help you improve the development effectiveness of your projects, programs, and strategies, as well as foster accountability and learning.
Marvin Taylor-Dormond
Director General
Independent Evaluation Department
Asian Development Bank
Today we’re talking about the concept of Independent Evaluation. What is Independent Evaluation? There are two words involved in this concept. One is the noun itself – evaluation. And the other is the adjective. What does independence mean in this context? Let us talk first about the first word – evaluation itself. Evaluation, as Michael Scriven simply put it, is a process to determine the worth, the merit, the significance of the object of evaluation. Evaluation is applied to objects, to personnel, to programs, to strategies, to projects. Evaluations are the products of this process. Evaluation, in this sense, is what it is known as a transdiscipline. And by the way, a very old transdiscipline. It’s a body of theories, of perspectives, of views, of approaches that is autonomous in itself, but that at the same time serves the needs of other disciplines. There are other transdisciplines, as Scriven call them, such as ethics, logics, communications, statistics. And as I just mentioned, evaluation falls into this category.
Now, how do we determine that worth, merit, and significance of the object of our evaluation? There are seven steps to an assessment of the worth, merit, significance of the object. The first step is to clearly identify and define the object. For instance, a car, and a bicycle, and an airplane. That means to define the boundaries of the object, the scope. The three of them dedicated to transportation but evaluation methods will differ in each of these cases.
The second step is to define the evaluation criteria. This refers to the areas of the object of evaluation to be assessed. For instance, if it’s a vehicle, maybe comfort and speed, cost. For personnel it may be on-time delivery of work of the staff, expectations on soft skills, etc. For program evaluation it may be relevance, effectiveness, efficiency, sustainability as we normally use in program evaluation.
The third step is to define the standards. So, the standards are the bars that the object of evaluation must pass to be considered great, fine, or under par. The standards have to be well understood by the owner of the object of evaluation, or by the person who is being subject to evaluation.
The fourth step is to define the methods to measure the ability of the object of evaluation to meet the standards defined by the evaluator. It could be, for instance, surveys. It could be data analysis of documents. It could be pure measurement with some gadgets that measure, for instance, gas consumption or speed. Or it could be, simply, perceptions. For instance, in the case of personnel. You observe how the staff is managing soft skills for instance. How it relates to other people?
The fifth step, this is a critical step of the process of evaluation, measure the degree of accomplishment of the object. And now it is time to compare with the standards that you have defined.
The sixth step of the process of evaluation is to sum up the assessment by criteria and to establish an overall assessment by summarizing the criteria.
And finally, the seventh step, analyze the data. Understand how your object of evaluation has performed. Why it has performed the way you have determined. And write up your evaluation.
Those are the seven steps for evaluation in general. Now we come to the second concept – of independence. Independence actually means to be free, to be protected from material threats to objectivity. And what are risk to objectivity? Well, they come in different forms. The threats to objectivity may take the form of organizational arrangements. For instance, the evaluator reports to the owner of the object of evaluation. Or that the owner of the object of evaluation determines the resources that evaluator can have to conduct this work. They also may be risk associated with conflict of interest. The evaluator may have been involved with producing the product. There may also be behavioral risk associated with biases of the evaluator. There are a number or risks apart from the ones a just mentioned. You will never be free of these risks completely because of the dynamic and contextual nature of the risk. But there are policies that can be set up against this risk to manage, for instance organizational issues. There may be operational guidelines to control issues related to conflict of interest. There may be training with respect to behavioral awareness. But the fundamental point here is that independence is the protection of the objective view of the evaluator.
There we go. Now we have the two concepts together. Independent Evaluation is the process of determining the value, the merit, the significance of an object of assessment in an objective manner.
Thank you very much and I hope these concepts are now clear in the minds of evaluators and non-evaluators as well.
Thank you for watching Evaluation Guru. We will be back with another episode on how to design, implement and manage effective evaluations. Till then, don’t forget to look up our social media accounts. Bye for now.
