Evaluation Guru: Evaluating Private Sector Operations

Transcript

Hello and welcome to Evaluation Guru. Through these series we will share with you how to design, implement and manage evaluations which in turn will guide your programs and projects towards development effectiveness, foster accountability and learning.

Nathan Subramaniam
Director
Independent Evaluation
Asian Development Bank

Hello everybody. Welcome to this new session. We are introducing a new serieson contributing to the knowledge baseon evaluations. Today's topic is evaluating private sector operations. And we're gonna be talking about how to evaluate private sector from an independent evaluation perspective. Independence in evaluation is absolutely crucial so as to ensure that there is proper accountability in this evaluation and there are no undue pressures in affecting the assessment of accountability. We all know private sector is an important component of worldwide economies. They're increasingly becoming a big share of all economies irrespective of their level of development. Private sector contributes considerably to employment, can improve productivity, and improve efficiencies. And it's crucial for us to be able to support private sector and all these economies and gather up all the benefits that we can get from contributions of private sector.

And hence it is important for the policymakers and the regulators who has to be able to create the proper enabling environment for private sector to thrive, and it is important also for evaluators to be able to look at private sector operations and identify areas where things can improve so that we can contribute to learnings and contribute to improvements in design and targeting of private sector operations so that a wider community across the entire society could benefit from private sector operations.

So how do we evaluate private sector?

In evaluating private sector there are important considerations that we need to take note of. One is it needs to be a market-based approach, secondly we need to make sure that we bring the accountability elements in the evaluation, and there should be a learning component. So in evaluating private sector we factor in all of these and we have come up with a harmonized approach across Multilateral Development Banks (MDBs). And we have zeroed in on four criteria for evaluating private sector. The first is development results. Second is additionality. Third is investment profitability. And the fourth is work quality. I'll go quickly into each of these criteria and give you a flavor of what we consider as part of each of these criteria when we evaluate private sector operations. In terms of development results we look at four subcriteria. The first is private sector development where we look at the in-company effects and the outside-company effects. So in a private sector operation we are supporting a particular company in a developing member country. And here, as part of this private sector development assessment, we look to see what is improved within the company and what are some of the demonstration and catalytic effects of these improvements on the wider economy. Second, we look to see what are the business success elements of this company. So we have supported this company, we wanna make sure that this company is thriving and is gonna be contributing to the economy over a long period of time. So we look to see what are the financial returns in terms of investments in this company. And we wanna make sure that those are healthy so that other private sector investors would also participate going forward. Thirdis to look at the economic contributions of this investment. Here we look to see what are the contributions this investment has made, in this particular company, to the wider economy. Examples of that could be improvements in capacities, skills. It could be contribution to taxes and so on. And finally we also look at the environmental and social impacts of this particular investment. We wanna make sure that the environmental concerns are all properly addressed, there is no harm that is done to the environment, and the social elements are also properly considered as part of this investment. So these four subcriteria together constitute the assessment of development results which is similar to what you would consider as development outcomes in a particular investment.

The next aspect of our assessment is additionality. Additionality is crucial for investments that are through the private sector. Additionality can be thought of as inputs and services, so as to contribute to development outcomes over the long term. And these inputs and services could come in the form of financial additionality such as in terms and conditions. Maybe an investment could happen with longer tenure for the loan that otherwise doesn’t exist in the market. That could be financial additionality. It could also be non-financial additionality such as institution building, corporate governance improvements at the company level or at the wider economy level that will help facilitate this investment. So together financial and non-financial additionality are combined to come up with an assessment of additionality. The third aspect that we look at is the investment profitability. And here we wanna look to see we are properly compensated for this investment and that we are not displacing the market through the use of public money for private purposes. And so we look to see the market comparables and we wanna look to see that the pricing that we’re coming up with for this transaction is commensurate with the market. And finally we look to see what is the work quality of this intervention. And this is important for us to come up with lessons so as to improve the design elements of private sector operations going forward. And we also consider the monitoring and implementation aspects of this particular intervention.

Key Challenges in evaluating private sector operations

Ok, so obviously there are somekey challenges that need to be overcoming evaluating private sector operations.

1. Cost of a robust monitoring and evaluation framework

The first biggest challenge that we all encounter in evaluating private sector operations is the cost of coming up with a robust monitoring evaluation framework. After all if something cannot be measured well, it cannot be tracked and reported well, we cannot really deal with it. So key to doing a good private sector operations is to come up with a robust results framework. So that we can design it ex-ante and the data that is required during the implementation is properly tracked and reported on. So that when we come in to evaluate this operation ex-post we have all the necessary ingredients for us to evaluate. And obviously there is cost associated in order to design a good monitoring and evaluation framework, and to track and report on this data. And some of the private sector players look at this and say this is something that is not attractive and so we won’t be competitive enough. And that needs to be overcome.

Profit vs Development

And related to this is the second challenge thatI wanna highlight, which is this broad perception that is there in the market that profitability and development outcomes do not go hand in hand. But based on our large body of evidence that we have gathered up in our previous evaluations which is over a hundred different evaluations we have done over the last decade, there seems to be a very good correlation between development outcomes and profitability. And they’re not mutually exclusive. And in fact good development outcomes often result in good profitable transactions. And this perception needs to change and that is the challenge that needs to be overcome with good communication. And hopefully modules like this can help alleviate, improve that perception.

Importance of communicating evaluation findings

So communicating results from evaluations is absolutely crucial. After all we don’t wanna do evaluations for the shelves. One of the key elements that should come out of the evaluation is lessons on what can be improved going forward. And so the various stakeholders that are impacted by this particular investment need to all benefit from the findings that come out of this evaluation. So there are immediate stakeholders within the bank who are the operations team who are executing this investment. And they need to benefit. They need to have their own set of lessons. And more importantly there are stakeholders on the ground which are the beneficiaries in the developing member countries who are the recipients of this investment who also need to be able to receive the lessons that are pertinent to them, so that they can get the best out of this investment. And also in between there are these sponsors and facilitators of this investment in all these countries that also need to receive their own lessons so that they can fine-tune and improve the efficiency of their own operations. And in our evaluations we pay important attention and focus to come up with appropriate lessons for the various stakeholders. And it is very very important as you identify these lessons, to communicate them to the wider audience, both within the bank and outside the bank so that the benefits can be shared and operations going forward can be improved.

Key take-aways from this module

1. Evaluation is crucial to find lessons

Alright, so I have two key take-aways from today’s module.

The first is evaluations are absolutely crucial in order to arrive at good lessons so as to improve future operations. And here we have looked at lessons, and I talked about this before, lessons that are pertinent not only to the bank operations but lessons that are important and useful from a beneficiary and other stakeholders’ point of view. And this means something that all evaluations need to keep in mind and be able to come up with lessons that are useful for the realistic goals.

2. Evaluation is crucial to assess development outcomes

The second key take away that I wanna give this part of this module is the importance that need to be given to assessing development outcomes. Private sector operations are important element of developing economies and they are increasingly becoming a bigger share. And especially during this COVID times, private sector plays a key role to alleviate pressures on fiscal burdens. And in order for private sector to be effective in contributing to all elements of society including addressing key ills of society such as inequality and poverty, it is important to focus on development outcomes. And evaluations should double down and come up with a clear robust criteria for assessing development outcomes. Thank you for watching Evaluation Guru.

We will be back with another episode on how to design, implement and manage effective evaluations. Till then, don’t forget to look up our social media accounts. Bye for now.