Practical Evaluation

Lessons Learnt – how should they look like? Sam Norgah, Feb, 2010

2011-07-19T19:21:00.000-07:00

Introduction

This document is based on contributions made by members of the African Evaluation Association (AfrEA) on the question, ‘how should lessons learnt look like?’

One of the key functions of evaluation is to generate a body of knowledge or experience relevant for improving future performances and enhancing quality. Often classified as ‘lessons learnt’ in an evaluation report, commissioners of evaluations refer to this section in a bid to draw on the key ‘lessons to be learned’ from the program/project through the evaluation
which has been carried out.

One of the challenges faced by commissioners of evaluations with regards to ‘lessons learnt’ is that evaluators often state the ‘obvious’ without reference to context and scalability. In fact, some of the lessons outlined in the evaluation reports are ‘common sense’ and ‘general knowledge’ which do not require an evaluation to unveil. Considering the amount and extent of resources committed to evaluations and the pressure on development practitioners to ensure accountability, demonstrate results of their intervention and scale up their interventions, emphasis on lessons drawn from programs has become very critical and in most cases, non-negotiable.

Extent of the problem

NGOs are not alone when it comes to concerns about the relevance of lessons drawn from evaluations. A study by the United Nations Environmental Program (UNEP) in 2007 showed that nearly 50% of evaluations conducted between 1999 and 2006 did not meet their ‘quality criteria’(Lessons learned – a platform for sharing knowledge, Special study paper). This staggering figure reported by an organisation like UNEP raises concerns about the scale and trend of this issue and it is therefore imperative for both commissioners of evaluations and evaluation practitioners to take steps to strengthen lessons drawn from evaluations and ensure clarity around those lessons.

According to opinions from both commissioners and practitioners, one of the main reasons for this state of affairs is the lack of clarity around what should be captured under lessons learnt. In most cases, the ToR which guides the evaluation is not explicit on this. Notwithstanding this
argument though, in my personal opinion, I think the major responsibility lies with the evaluation consultant (as an expert) to be able to draw the line between ‘general knowledge and key lessons to be learned’ from the program/project. This in my opinion raises the question of capacity first and foremost among our practitioners and secondly among the
commissioners – a sentiment shared by many of the contributors on this issue.

Key criteria for identifying and capturing lessons:

Three sets of criteria emerged from the various contributions; I have stated them below. The key thing is that there is the need for clarity and agreement on what needs to be captured - this needs to be explicitly stated in the ToR.

1. According to UNEP, the quality criteria for assessing lessons learnt are:

(i) Concisely capture the context from which the lessons are derived

(ii)The lessons should be applicable in different contexts (generic) and should have a clear ‘application domain’ and identify target users.

(iii) The lessons should suggest a prescription and should guide action

2. There are five levels of lessons which can be drawn from any evaluation (Clarke and Rollo 2001); these are: (i) general Information, (ii) explicit knowledge, (iii) tacit knowledge, (iv) insight and (v) wisdom.

It however appears that for most evaluations what evaluators have
focused on is the first level – general information. This was confirmed by the UNEP study and contributions from other evaluation commissioners.
According to one commentator the levels that matter the most are the last two, provided they are supported by strong and rigorous evidence from the evaluation methods and findings. The essence of all evaluation processes is to draw relevant lessons - there is the need for a shift from gauging what has taken place in the past for its own sake and stating the obvious. The
challenge then is how to formulate lessons as research questions.

3. According to another contributor, two principles which have helped in articulating lessons are:

(i)Clarifying the intended use and intended users – at the time of agreeing and finalising the ToR and planned M&E processes, it is important to clarify the outcomes, learning points and the key audience

(ii) Develop a structured set of learning points and recommendations using appreciative inquiry. A staff of an international NGOs alluded to the challenge of clearly articulated lessons and suggested that presenting evaluation feedback through workshops with stakeholders is critical
forum which allows for reflection, documentation and dissemination.

Conclusion

Drawing relevant lessons from evaluations is a challenge faced by both commissioners of evaluation and practitioners alike. It was generally agreed that capacity is partly to be blamed for this state of affairs. This trend calls for standardisation of processes and a renewed call for
accreditation of the profession.

Overall, it’s been an interesting and insightful discussion – thanks to all who contributed to this topic and for sharing your experiences and lessons

Lessons Learnt

2010-02-08T06:01:00.000-08:00

I've been struggling to understand what 'Consultants' mean by 'lessons learnt' whenever they conduct evaluations for us. To a large extent, what is captured as lessons are just obvious statements and observations; please see examples below.

Child-Centred Community Development (CCCD): Lessons Learned

1. Orchestrating the evolution of CCCD as a rights-based approach requires continuous support and consistent leadership and communication.
2. Most Plan-supported communities and partners have not been adequately engaged in a dialogue to understand the concepts and implications of Plan’s CCCD approach, making effective partnership a challenge.
3. A stronger and more regular engagement with partners’ and sub-contractors’ activities is needed to better ensure that project objectives and targets are met.
4. Identifying and engaging with grass-roots civil society organizations and groups demands a proper mapping of such organizations and groups, and going beyond working exclusively with established local leaders.
5. More effort needs to be made to establish Plan Uganda in national-level networks and relevant working groups for purposes of advocacy, and positioning Plan for influencing policies and recognition as a key actor in development.

As you can see, most of the statements above are very obvious. Can someone please help me understand what lessons learnt should capture? Your examples will be appreciated.

Sam

Template for the Utilisation of Evaluation findings

2009-04-16T09:47:00.000-07:00

General Introduction

Optimum use evaluation findings is a real challenge for most Organisation; this challenge keeps coming up in most evaluation forums. To move a step towards improving the use of evaluation findings, I have put together, a brief template on how to encourage Organisations to use evaluation findings and also to hold themselves accountable for the use/non-use of the findings. This template though very simplistic has been tried in our Organisation and it seems to be working. This template can be used for forward planning or as a template for assessing how evaluation results have been used. Your comment on this tool will be appreciated, feel free to use/adapt this tool

Evaluation Utilisation Template

1. Introduction: The brief introduction should include information on the following (among others); the title of evaluation/study/research, the date on which the evaluation/study/research was conducted, name of consultant (if internal, indicate the name of the lead person) and the purpose for the evaluation (including the person who commissioned the evaluation)

2. How was (will) the result of the evaluation/study/research (be) used? It will be useful to indicate how the evaluation findings/results will be used or (was used). Some of the key uses might include; (a) Program improvement - what has changed or what is going to change as a result of the findings, (b) use for Advocacy - indicate which policy has been influenced or will be influenced, (c) for future program planning, (d) donor requirement (e) Publications (where have this been published/where will it be published). If the findings have not been used, please explain why?

3. How was the findings shared with other stakeholder? Since most Organisations work in partnerships, it will be useful to indicate in the report, which stakeholders were involved in sharing the results and what format was used

4. Any follow-up action required? The crust of the evaluation is the utilisation of the findings. Closely linked to section 2 above, is the question of follow-up. It will be useful to indicate all the follow-up actions and who will be involved. It will also be useful to indicate whether the findings will impact on other programs or interventions

5. Action planning: Very bold and specific statements on the way forward.

What are your thoughts? Cheers

Creating a Learning Culture

2009-01-28T02:56:00.000-08:00

Closely linked to the issues of utilisation of evaluation reports/findings is another pertinent challenge: how can we create a learning culture within our Organisations and among our clients? Evaluation findings are suppose to enhance and promote program quality and learning. However we often find organisations implementing similar projects or interventions even though evaluation findings point otherwise. I have seen NGOs embark on non-sustainable interventions over and over even though results from their own evaluations show that such interventions do not work. How can we as evaluators influence decisions based on our findings? Does our work end once we've submitted our reports or could we do more to support our clients to shape their program direction? Your thoughts and experiences.

Utilising evaluation findings

2009-01-19T08:46:00.000-08:00

Most development Organisations spend a lot of resources (financial, time and human) to conduct evaluations and other researches however there are compelling evidence to show that evaluation results are not fully utilised. As development practitioners and Evaluators, how can we contribute to the discussion on 'utilisation of evaluation findings?' Please share your thoughts and experiences