Evaluating Social Change and Communication For Social Change: New Perspectives

This 25-page essay explores strategies for assessing and demonstrating the impact of social change and communication for social change processes. Using the meeting of HIV/AIDS Implementers in Kampala, Uganda, in June 2008 as a launching point, Ailish Byrne lays out the challenges and diverse factors impacting monitoring and evaluation (M&E) in the field of social change communication (SCC). She raises questions regarding many current M&E practices and their implications for development.

Byrne discusses the SCC panel she was on at the Kampala meeting, stressing that her fellow panelists were united by their belief in the power of communication to address the social drivers of HIV/AIDS - including socio-cultural norms that impact an individual's behaviour and degree of choice. The Joint United Nations Programme on HIV/AIDS (UNAIDS) Social Change Communication working group has defined SCC for AIDS as "the strategic use of advocacy, communication and social mobilisation to systematically facilitate and accelerate change in the underlying drivers of HIV risk, vulnerability and impact". A key focus of the discussion was the limitations inherent in HIV/AIDS programmes and prevention strategies that rely on individual behavioural change approaches, which pay insufficient attention to social and cultural norms and contexts. In addressing this communication challenge, the participants emphasised the importance of developing and sustaining alliances and partnerships, of sustained capacity development and support, of sharing results and lessons learned, and of strengthening and actively supporting the M&E of SCC. Byrne notes that the process of developing the session, as well as feedback afterwards, reaffirmed the need for more concrete examples of the impact of SCC.

This plea for sound evaluation and demonstration of the value of SCC approaches raises, for Byrne, fundamental questions about why effectively demonstrating the impact of SCC appears so challenging and difficult to achieve in practice. Questions include: why it remains so difficult to negotiate and fund innovative evaluation practice, why there appears such resistance to changing dominant M&E practice at senior levels, and who needs to change. Byrne asks: Who makes influential decisions and on what basis? What constitutes appropriate data? Who defines this and whose interests are ultimately served? Underlying the discussion is a plea to broaden and complement (rather than replace) current M&E practice.

As a way into these questions, Byrne examines how it came to be that randomised control trials (RCTs) are construed as the gold standard for evaluation. Yet such methodologies, as one commentator Byrne quotes here argues, "are useful only for discrete and relatively simple interventions geared to precise and measurable objectives and characterised by well-defined 'treatments' that remain constant throughout program implementation" - characteristics that do not apply to the complex, multi-faceted, and largely unpredictable processes of social development, social change, or SCC. Byrne explores the idea that the neglect of qualitative approaches - those that foreground human agency, interpretation, and complexity - "illustrates how powerful and non-neutral forces actively perpetuate and impose the status quo, often for dubious reasons, while actively inhibiting innovation and healthy experimentation in the process."

Noting that the "costs of failing to appreciate the significance of socio-cultural factors and norms like multiple concurrent partnerships, circumcision, gender violence and inequities to HIV prevention, are increasingly realised", Byrne cites Warren Feek of The Communication Initiative. Feek has referred to the Mexico XVII International AIDS Conference in August 2008 as the "social conference" because of the focus on social drivers, social complexity, social change, social mobilisation, social movements, social stigma, socio-economies, socio-cultural factors, and social phenomena. In short, Byrne argues that there is broad(ening) recognition of the role communication clearly has to play and growing calls for broader social campaigns or social movements to address such issues.

How to heed these calls? To understand this puzzle, Byrne examines characteristics of social change and implications for its evaluation. She cites several models (e.g., the Soul City model of social change) that illustrate why a strategic, multi-level, multi-component response is needed. In particular, research into complex adaptive systems (CASs) highlights the difficulty of attributing change to any one intervention. Byrne is thus moved to turn to what complexity thinking can offer - an inclusive, overarching approach to understanding societies as complex systems and subsystems, all of which interact with each other. One defining characteristic is an overriding concern with social processes rather than social structures. Complexity theory highlights the links between context-specific social processes, standards, norms, and values and, therefore, the danger of assuming replicability or scale-up. Complexity theory, in short, encompasses an understanding of change that privileges networks, relationships, and process.

If, to be effective, an evaluation programme must match the dynamics of the system to which it is applied, characteristic behaviours of CASs must be reflected in their evaluation. Byrne cites literature that highlights the implications of the key dynamics of CASs:

1. Dynamic - Openness to external influences means a CAS is constantly changing - and not in a smooth, predictable pattern. Thus, there is need for flexibility, responsiveness, and openness to what emerges.
2. Massively entangled - Complex interrelationships mean incorporating multiple strategies, cycle times, time horizons, dimensions, and informants. Thus, a variety of data must be collected.
3. Transformative - The CAS transforms and is transformed over time. Effective evaluations will respond by making evaluation part of the intervention, and by involving as many members of the system as possible in the evaluation design.
4. Emergent - System-wide behaviours of a CAS emerge over time, so it is important to track emergent patterns and pattern changes over time.
5. Scale-independent - CAS incorporates many levels, so evaluation must incorporate both micro- and macro- patterns and structures.

What are the implications of this analysis for SCC? Byrne here examines the thinking of Paolo Mefalopulos on evaluating participatory development and communication, where "dialogic communication" is the norm. While more traditional communication interventions associated with a monologic mode can generally be assessed in ways that sit more comfortably with typical organisational cultures and M&E systems, dialogic communication is harder to evaluate. Anecdotal evidence, impressionistic evidence, and evidence of problems or failure resulting from a lack of dialogue with stakeholders are often more appropriate to the evaluation of participatory communication, but pose particular challenges.

Byrne indicates that such perspectives are more in keeping with broader understandings of development and social change that are fuelling greater interest in innovative ways of monitoring and evaluating social change. She argues that it is senior stakeholders who have been most reluctant to grasp the implications of these shifts in thinking - in part because, as one thinker Byrne cites explicitly argues, participation challenges patriarchy and the power and security of many "uppers" (more powerful stakeholders). An alternative way of framing international aid "advocates a political rather than technical understanding of accountability, with a focus on struggle for voice and justice. There have been significant shifts to develop and strengthen evaluation methodologies and methods...[that] validate and legitimise new experts (in particular lay- as opposed to just professional knowledge), and new types of knowledge (stemming from lived experience, as well as that which is learned)....Fundamentally such perspectives entail taking evaluation out of the realm of 'experts' and capitalising on local knowledge, strengths, experience and abilities." Byrne discusses in depth the role of critical dialogue, the sense in which evaluation is fundamentally educational, and the relational nature of this approach.

Byrne moves on to examine particular elements of participatory monitoring and evaluation (PM&E), "a significant and growing body of comparable methodologies that foreground communication and dialogue...[and which] highlight how evaluation can itself encompass a social change process and can be a stimulant and catalyst for social change." Indicators, however, remain the dominant mode of evaluation, and Byrne spells out their limitations. To cite one example, "They do not effectively capture qualitative changes in people's lives and experience, which results in limited information that often misses great richness and depth that lies at the heart of SCC." This weakness, and the others she lists, can, she says, be somewhat mitigated by a participatory approach to indicator selection and use. However, Byrne advocates an approach to evaluation that goes far beyond indicators. The approaches she discusses include: the Most Significant Change (MSC) approach, a methodology explicitly based on systematic story collection, analysis, and filtering; and Outcome Mapping (OM), which explicitly recognises that a programme does not operate in isolation from wider external factors and highlights, therefore, that planners and evaluators should not proceed as if it does.

In light of this discussion - and of Byrne's observation that any participatory approach, by definition, is emergent and can always be strengthened - she offers some specific recommendations for practical ways forward:

• Generate practical ideas and share inspiring examples, which involves investing in: concrete efforts to systematise and review respective benefits and limitations of different grounded case studies; capacity development efforts for social change organisations on how to assess social change in ways that recognise core non-negotiable principles and purposes; peer support opportunities to request and receive support with assessment and learning processes; and seeding experimentation and detailed documentation of processes.
• To achieve credibility and effectiveness, evaluation processes should combine independent evaluation, self-evaluation, and full involvement of citizens.
• Combine methods as appropriate and question the bases on which particular methods are selected.
• Ensure flexibility in evaluation design and practice - this is part and parcel of designing evaluation processes closest to where they will be used).
• Look actively for community strengths, assets and achievements, rather than being focused on problems and weaknesses.
• Seek to redress power imbalances through participatory approaches to evaluation that actively foster equity and voice, as well as collective reflection and learning for improvement.
• Donors should consider the impact of biases and underlying assumptions on their evaluation policies and practice.
• Intermediaries have roles to play in helping to ensure quality evaluations by strengthening capacity across the board, seeding experimentation, proposing alternatives to complement and strengthen current practice, and dialoguing with donors and others.

In concluding, Byrne acknowledges that "[t]here is a long tradition of those opposed to change using rigour to critique participatory methods. It begs questions about which partner defines what 'rigour' is, whether information can be 'good enough to move forward' or must be 'perfect' (according to whom?), and whose norms count....Powerful forces continue to preserve the status quo and to undermine alternatives to assessing social change, implicitly and explicitly....Do not underestimate the influence of the dominant paradigm and evaluation 'machinery' that is witnessed in legitimised experts, capacity development and training opportunities and the resources available or not for particular evaluation methodologies and methods. As Robert Chambers has repeatedly emphasised, questions must be asked about unlearning and relearning required, and about who needs to learn. The importance of openness to change and to a degree of risk-taking cannot be overstated."