Review of 16 Reflect Evaluations

This 59-page paper, published by ActionAid, consolidates evidence, learning, and evaluation methodologies from 16 evaluations of the "Reflect" process, a participatory approach to adult learning and social change fusing theories of Paulo Freire with participatory rural appraisal methodologies. Reflect seeks to link the literacy acquisition process with individual and community empowerment, strengthening the capacity of people, particularly women, to secure their basic rights. According to the publishers, Reflect programmes operate in diverse contexts, and approaches to evaluation have been equally diverse - making it difficult to consolidate. This review of 16 evaluations was conducted in order to draw out learning from existing evaluations to feed into a new evaluation framework. The report contends that it is clear from the analysis that many innovations are taking place, particularly in terms of the design of evaluation tools and careful consideration of which processes are appropriate for each context. However, greater standardisation is needed in order to be able to clearly assess the impact on communities and individuals.

The review is divided into two parts:

Contributions of reflect to literacy

Section one of the report examines evidence linking Reflect to literacy and other development outcomes. The researchers found the following:

• The evaluations reveal that Reflect programmes are associated with a wide range of development outcomes, including literacy development.
• It is important for programmes to have a clear understanding of the literacy context and of literacy in its broader sense as a means of framing and developing a programme. While many programmes acknowledge the wider context of literacy, this is often in an implicit and poorly developed way.
• Focus on literacy as being about "basic literacy" may mean that programmes lack focus on sustaining literacy and the development of literacy and post-literacy habits that ensure participants continue to develop their skills.
• Some programmes appear to see a dichotomy between literacy acquisition and the achievement of other development goals such as empowerment and community action. However, evidence suggests that the two are integral parts of a process and that the development of one has a positive impact on the achievement of the other.
• Many Reflect programmes seem to have a misconception about the use of supplementary reading materials in Reflect, leading to missed opportunities to introduce external literacy materials into Reflect circles.
• The majority of Reflect programmes are making a positive contribution to literacy development. Even programmes that focus on empowerment and do not involve any direct teaching of literacy can still have a positive impact on literacy acquisition and skills development. It is important to raise awareness of these diverse literacy outcomes so that literacy benefits beyond traditional teaching of basic literacy can be better understood and factored more consciously into the design of programmes.

The evaluation process

Section Two of the report focuses on analysis of the evaluation process. The section concludes that while many innovations are taking place, particularly in terms of the design of evaluation tools, the evaluations are by and large unclear on issues such as conceptualisation of literacy and the literacy environment, as well as purpose and ownership of the evaluation. The research found the following:

• The evaluation report is only one small part of the process, often written by just one person, but as the only archived documentation it can be seen to represent the whole evaluation. It is therefore important to make it clear who the author of the report is (and his or her relationship to the evaluation and the Reflect programme), who the report is for, and how the process of writing the report was conducted (i.e. were there meetings/consultations to inform drafts and if so, who participated?)
• It is important to clearly state the assumptions of the evaluation, for example, conceptual underpinnings (definition of literacy and the literacy environment), purposes of the evaluation (summative, formative, and capacity-building), ownership of the evaluation (who funds and implements and who the evaluation is for).
• Detailed attention should be given to the evaluation methodology. The different phases and participation in each phase should be carefully considered.
• Baseline data are imperative in order to conduct a legitimate evaluation.
• Careful consideration of timing and timetabling should inform the planning of any evaluation, as should the sampling of Circles when all can not be visited.
• Power issues should be kept in mind and discussed at different stages throughout the evaluation process. Ways of mitigating power relations should also be discussed, especially where power is a barrier to participation and to obtaining legitimate findings.

Overall, evidence suggests that although the majority of programmes cite literacy development as a goal, they tend to focus on basic literacy skills and their functional use in the community without a clear idea of the need to develop post-literacy habits. At the same time, researchers found the evaluation reports to be mostly unclear on conceptual issues and approaches. They also mention that, while there is a need to retain the uniqueness of the context and objectives of each Reflect programme, there is also a need for a similarly unique response to evaluation, which can be gathered in a cohesive manner.