1. bookVolumen 36 (2015): Edición s1 (May 2015)
Detalles de la revista
License
Formato
Revista
eISSN
2001-5119
Primera edición
01 Mar 2013
Calendario de la edición
2 veces al año
Idiomas
Inglés
Acceso abierto

Tensions, Challenges and Issues in Evaluating Communication for Development: Findings from Recent Research and Strategies for Sustainable Outcomes

Publicado en línea: 07 Jul 2020
Volumen & Edición: Volumen 36 (2015) - Edición s1 (May 2015)
Páginas: 25 - 39
Detalles de la revista
License
Formato
Revista
eISSN
2001-5119
Primera edición
01 Mar 2013
Calendario de la edición
2 veces al año
Idiomas
Inglés
Introduction

Communication for development (C4D) is widely seen as important to achieving sustainable development and social change (Jallov 2012; Quarry and Ramirez 2009; Servaes et al. 2012). However, Feek and Morry (2009) found that C4D lacks central status in policy, strategy and planning, lacks impact data, skilled C4D staff and dedicated funding, and that corporate communications were prioritised. Other long-term research highlights a recurring problem with decision makers in development organisations not appreciating what C4D means, or its important role in development (Lennie and Tacchi 2013).

The United Nations (UN) has attempted to raise the profile of C4D over the past 25 years through various strategies for mainstreaming C4D and improving the evaluation of C4D. Moving C4D up the development agenda depends on finding more effective ways to demonstrate the impacts and contributions of C4D to development (Lennie and Tacchi 2013). However, Puddephatt et al. (2009) concluded that there is no systemic use of monitoring and evaluation (M&E) to demonstrate C4D impact among UN agencies. C4D addresses complex problems such as reducing poverty and gender discrimination, so assessing its impacts and outcomes is equally complex. Our recent research (Lennie and Tacchi 2011, 2013) identified numerous contextual, structural and institutional challenges, issues and barriers, including a lack of appreciation, funding and support for alternative evaluation approaches that are increasingly seen as more appropriate for the evaluation of C4D and other complex social change initiatives. We also identified significant tensions, challenges and contradictions in assessing the impacts of C4D. They include problems with attribution, and pressure to demonstrate impacts within unrealistic timeframes, using inappropriate evaluation approaches.

This paper explores these and other tensions, challenges and issues related to the effective evaluation of C4D, drawing on our consultations with C4D and evaluation experts and other recent research. We propose a range of strategies to overcome these challenges and achieve more sustainable C4D outcomes then consider some policy implications.

Background to the Article

This article is informed by outcomes from a 2010 project we conducted in collaboration with various UN agencies which developed an initial version of the UN Inter-agency Resource Pack on Research, Monitoring and Evaluation in C4D for use by the UN and its partners (Lennie and Tacchi 2011). Our research included a wide-ranging literature review and consultations with a 15 member Expert Panel from around the world and 11 C4D Focal Points or M&E specialists in seven UN agencies, funds or other bodies. We conducted interviews and a detailed online survey, and gathered feedback on draft principles for evaluating C4D and on a draft report that included an initial evaluation framework.

Based on this and earlier work, we later developed an overarching framework for evaluating C4D (Lennie and Tacchi 2013; Tacchi and Lennie 2014), which we will briefly outline later on. A key research project that informed this work was Assessing Communication for Social Change (AC4SC). This was undertaken in collaboration with the NGO Equal Access Nepal (EAN). It developed a participatory methodology and M&E systems and processes to assess the impacts of C4D radio programs made by EAN. This provided significant learnings about the challenges of developing and implementing participatory approaches and rigorous M&E systems to evaluate the impacts of C4D initiatives and build evaluation capacities in a complex and challenging development context (see Tacchi et al. 2013; Lennie et al. 2012).

The Current Development Evaluation Context: Implications for Evaluating C4D

The increasing complexity of the development context since the 1990s poses significant new theoretical and methodological challenges for development evaluation, including the evaluation of C4D. A broader vision of development centred on the Millennium Development Goals has emerged, with a greater emphasis on effectiveness, targets, and partnerships between donors and aid recipients, following the 2005 Paris Declaration on Aid Effectiveness (Conlin and Stirrat 2008: 196). However, many development initiatives, including C4D programs, have emergent goals that are negotiated in dialogue with stakeholders rather than having pre-determined outcomes. They are longer-term, high risk programs and their impacts are often difficult to evaluate using standardised or established tools (Stern et al. 2012: 11).

This situation has led to growing tensions between results-based (accountability) approaches and emerging learning-based (improvement and effectiveness) approaches to evaluating development interventions (Armytage 2011). In the former, impacts of complex interventions are often reduced to simple, cause-effect processes, using logframes, indicators that pre-determine impacts and outcomes, and methods that often prioritise quantitative data. In contrast, participatory, systems and complexity-based approaches understand social change as emergent, unpredictable, unknowable in advance, something to learn from and adapt to, that may have contradictory or negative outcomes (Burns 2007; Ramalingam et al. 2008). These approaches use a range of flexible techniques and mixed methods to better understand systems, networks and inter-relationships and the wider, often subtle, ripple effects that are important to long-term change. They appreciate the importance of community participation, dialogue and ownership, two-way communication and feedback loops, and attending to gender and power relations and local social and cultural norms to achieving sustainable development and transformational social change (Burns 2007; Quarry and Ramirez 2009).

A good example of this new evaluation approach is outcome mapping (Earl et al. 2001), which has shifted from a focus on assessing the impacts of a program (defined as changes in state such as reduced conflict) towards changes in behaviours, relationships, actions and activities of people, groups and organisations. This approach focusses on the more subtle changes that nevertheless “are clearly within a programme's sphere of influence” (Earl et al. 2001: 10).

Contentious debates are emerging about the challenges and limitations of increasingly dominant results-based management (RBM) approaches (Conlin and Stirrat 2008). Armytage (2011: 274) suggests that until the challenges of development evaluation are addressed “there will remain a marked ‘evaluation gap’ between the theory and rhetoric of the Paris Principles on Aid Effectiveness and the real world of development evaluation practice”. RBM and other upward accountability approaches clearly have many limitations for evaluating C4D. This has significant implications for better policies and practices related to the planning, implementation and evaluation of C4D initiatives.

Tensions, Challenges and Issues in Evaluating C4D

Our research has identified many complex challenges, issues, tensions and contradictions in evaluating C4D. They are contextual, structural, institutional and organisational and affect the long-term sustainability and success of social change and development initiatives. Balit (2010a: 6) points out that both development and communication are basically political and this is why “political will to put into practice on the part of governments and local authorities is often lacking. After all, enabling poor communities to participate directly challenges existing power structures”. A similar argument can be made about the participation of a wide range of people in the evaluation of C4D.

Contextual Challenges and Issues

Numerous complex social, economic, political, cultural, environmental and technological factors affect development and C4D, including issues of power, gender and other differences. The context of C4D includes communities, organisations and institutions as well as geography, history, culture, political and economic systems, rapidly changing information and communication technologies, and media systems and institutions. The evaluation of C4D is also affected by the funding rules and requirements of donor organisations and the attitudes and practices of development managers and decision-makers.

There are several specific challenges in undertaking evaluations in developing countries, associated with geographic, communication and cultural barriers, local political issues and other factors. They can significantly affect communication among evaluation participants and travel to research sites, making field research and data collection more time consuming and difficult. We experienced major communication and travel problems in the AC4SC project due to the country's wide cultural and linguistic diversity, high mountain terrain and poor roads, and limited internet access outside the Kathmandu Valley. Ongoing political instability and discontent in Nepal frequently involved strikes that disrupted the transport network. These problems greatly affected field research work and participatory capacity development activities, which involved program production and M&E staff from EAN and a network of community researchers based in various regions of Nepal.

Country and Institutional Level Challenges

Several significant obstacles have affected the development and implementation of the UN's C4D advocacy strategy, including its evaluation and capacity development strategies. Balit (2010a: 4) points out that C4D is a social process based on dialogue, it is a “soft and social science that has to do with listening, building trust and respecting local cultures – not easy concepts to understand for policy makers and programme managers with a background in hard sciences”. This means that quantitative and linear evaluation and planning approaches tend to dominate. However, as Balit (2010b: 2) notes, “counting and hard data cannot truly capture the complexity of social change processes over longer periods of time”. Our consultations found that some UN agencies emphasise the use of quantitative approaches which are unlikely to provide the most meaningful and useful data on C4D impacts. One of the Expert Panel commented:

A key issue underlying the challenges and difficulties is that the M&E of C4D (like much other development) is typically approached in a vertical, non-integrated manner, rather than being an integral part of programmes. An add on, for “M&E experts”. This reinforces the tendency towards top-down, “expert driven” approaches and actively works against participatory approaches

(skills for which the former do not typically have).

Many organisational challenges also affect the sustainability and effectiveness of the evaluation of C4D. One challenge in planning and conducting evaluations of C4D was “The lack of co-ordination between central HQ policy staff who want evaluations and field staff for whom evaluation is an irritation”. Along with Puddephatt et al. (2009), participants in our consultations for the UN Inter-agency Resource Pack emphasised the need for a long-term, sustained focus on capacity development in evaluation for staff at all levels. However, they suggested that without the understanding, funding, support and commitment of senior UN managers and donors, improvements to capacity and moves towards greater use of more innovative and participatory approaches and methods are likely to be less successful.

Attitudes and Policies of Funders and Management

Senior managers and funders were seen as lacking an appreciation of the value and importance of both C4D and evaluation, and tended not to support the use of more innovative or participatory approaches. One Expert Panel member identified the following as a key challenge:

The assumptions and biases of funders/those commissioning research and evaluation, combined with a lack of openness to less mainstream, more innovative, less prescriptive and predictable approaches. Both conceptually and in terms of resourcing these processes, an unquestioning “more of the same” is all too commonplace, regardless of the suitability and fit with the aims of and values underlying the particular programme involved.

Such assumptions are reflected in the lack of adequate funding and resources provided for the evaluation of C4D. Byrne (2008: 4) highlights difficulties with funding innovative evaluation practice in C4D and the frustrations of many at the field level with having to fit their achievements into externally imposed “SMART” objectives and logframes. Balit (2010a) also points to the problem of applying participatory processes within the rigid timeframes of logframes and RBM. Another issue is that evaluation studies often highlight successful C4D initiatives rather than those which were less successful but could provide valuable learnings, and lack of reporting on the long-term effects of communication programs (Puddephatt et al. 2009).

Challenges in Conceptualising, Managing and Planning the Evaluation of C4D

Analysis of survey responses from our UN Inter-agency Resource Pack consultations identified a wide range of challenges in conceptualising, managing and planning the evaluation of C4D, some of which we have already noted:

Insufficient funding, time and resources: Comments on this included:

Resources needed for research, if available (which they are usually not) would be disproportionate to the scale of the project/programme.

Under resourcing the effort, expecting impact results from what is really just “a drop in the ocean” case study.

One UN respondent listed her most important challenge as “Finding the time to design evaluations for diverse programmes, where each requires specialised analysis”. In addition, there was often pressure to “prove” results within a certain timeframe.

Low levels of skills, capacity, understanding or awareness of research and evaluation and social change: Comments included:

Uneven understanding of behaviour and social change.

Few skilled practitioners in many countries to conduct research, monitoring and evaluation of C4D.

Weak capacity for research and evaluation ... and inadequate resources to strengthen capacity at all levels, over a realistic timeframe.

Lack of capacity to design and implement research and evaluation, and lack of useful indicators or baseline data: Issues included:

Weak design of indicators, baseline information, and conceptual approach to assessing impact at start of implementation.

Evaluation is not really conceptualised at the beginning of programmes.

Diffuse, long-term and hard-to-measure results expected from our projects and programmes.

Indicators are too difficult for field or local staff to apply.

Lack of importance and value given to research and evaluation of C4D: Challenges identified included:

Convincing decision-makers and project managers that R,M&E of C4D is important.

Low level of realisation among partners of importance and value of R,M&E for C4D.

Lack of interest among programme staff, governments, other stakeholders in activities and use of evaluation results.

Attitudes to evaluation approaches, methods and processes: Challenges identified here indicated problems with the dominance of quantitative methodologies and with giving scant attention to deeper evaluation issues. Comments included:

To convince the contractor that quantitative methodologies will not provide the necessary information on how peoples’ lives changed. Only qualitative methodologies which allow people to participate and speak can provide quality information about social change.

The apparent obsession with methods and tools, to the neglect of deeper, fundamental questions like: Who is the evaluation for? What is it for? Who are the intended users of the evaluation? What are the intended uses? How will the process itself empower those involved and strengthen wider communication for development processes?

Challenges in Assessing the Impacts and Outcomes of C4D

Although impact studies have in the past been quite rare in development, since they are usually resource and time intensive, they are now high on the development agenda. Inagaki (2007) identified a lack of published reports on high quality impact assessments of C4D. Some of the difficulties in demonstrating the impacts of C4D were aptly summarised by one of the Expert Panel members in our consultations:

Impact is a holy grail, it requires considerable funding and effort to gain credible results because communication impact is challenging. It is not counting latrines that have been built, it is about assessing changes in how people think and respond to issues and contexts and this can be impacted by many variables.

Souter (2008: 181) argues that impact assessment of information and communications for development (ICD) programs requires “sustained commitment on the part of implementing agencies, from project design through to project completion and beyond”. He suggests that donors need to understand and be willing to recognise “that unexpected and even negative impacts need to be identified and understood; and that impact assessment is not about validation of past decisions but about the improvement of those that will be made in future” (Souter, 2008: 181).

A summary of the key challenges, issues and tensions in assessing the outcomes and impacts of C4D is presented in Table 1.

Tensions between Dominant and Alternative Approaches to Assessing the Outcomes of C4D

Dominant approaches Alternative approaches Tensions and issues
Dominance of instrumental, upward accountability-based approaches that focus on proving impacts, using linear cause-effect logic and formal reporting of results. Alternative approaches are not adequately resourced or supported and are often critiqued for lacking ‘objectivity’, ‘rigour’ and ‘validity’. Flexible, holistic interdisciplinary approach based on ongoing learning, improvement and understanding. Takes the complexity of social change and the particular context into account and focuses on outcomes that an initiative can realistically influence. Demonstrating the impact of C4D is complex and difficult. Dominant approaches discourage ownership of the evaluation process and learning from evaluation. Results are often biased towards positive outcomes, failures are not captured or learned from, and evaluations are not independent from donor influences.
Pressure to produce short-term results within rigid and unrealistic timeframes. This results in a focus on more tangible, short-term changes that are not good indicators of long-term social change. Seen as more important to focus on progress towards long-term social change and the contribution of C4D. This is a more realistic measure of effectiveness and provides practical recommendations for the implementation of policies and initiatives. Longitudinal studies are required but they are costly and one of the most difficult challenges in evaluation. Donors are reluctant to fund them. This means that there is a lack of strong evidence on which to build C4D research, which fuels scepticism.
Attribution Problems

Attribution is considered the “central problem” in impact evaluation (Leeuw and Vaessen 2009: 21). It is a key problem in assessing the impacts of C4D compared to some other development initiatives such as polio eradication programs where it can be easier to isolate changes in rates of the disease in a particular population. The processes and effects of communication can be difficult to measure. Balit (2010b: 1) suggests that in some cases we can think about measuring changes in “knowledge, behaviour, attitudes and access and use of services”. Yet the problem of attribution remains, given the difficulty of attributing causality.

Causality is complex, and change is likely to be due to a whole range of factors, which in turn act on each other. Different factors may become relevant over time. It is often quite difficult to track and isolate those related to C4D. This is, in part, due to C4D often being a component of a larger development initiative that is usually undertaken in collaboration with a number of partner organisations and involves a range of media and community-based activities. This presents a particularly difficult challenge, because of the politics of aid, which means that implementing agencies are “often tempted to claim credit for impacts because that is what those they are accountable to want to hear” (Souter 2008: 162). The complexity of assessing the impacts of C4D is highlighted by Inagaki (2007: 34–35):

... general categories such as mass media and interpersonal communication can potentially conceal varying effects among specific channels within each mode, such as one-to-one interpersonal contacts versus group discussion, broadcast media versus printed materials ... different communication channels interact with one another, and this interaction can form a complex network of communication effects encompassing multiple, direct and indirect paths of influence. When measured alone a mass media message may have negligible direct impacts, but the same message can have significantly greater impacts when mediated through other channels of communication, such as interpersonal communication and group communication.

The value of conventional evaluation approaches that are based on a program remaining static during the evaluation process clearly need to be weighed against the benefits of giving the freedom and flexibility to C4D initiatives to continually adapt and respond to changing ideas, contexts and environments and continuous feedback. The concept of cause and effect and causal relationships is not very useful here.

Timeframe Issues

Our research identified unrealistic demands, targets and timeframes for the impact assessment process, with donors expecting to see measurable results from C4D initiatives in an unreasonably short timeframe, most likely determined through measurable pre-set indicators. This can lead to the creation of “results” that may have little connection with activities on the ground.

If social change is understood as an emergent, ongoing and complex process, it becomes very difficult to understand and demonstrate the impact of a C4D initiative through measurable pre-set indicators within a short timeframe. Yet impact assessment is usually undertaken immediately after the end of a project's implementation. Social change is ongoing; outcomes of interventions often lie in the future, beyond the immediate project (Souter 2008). The main issues here are the timeframe of development funding, and reporting requirements based on dominant upward-accountability evaluation approaches. These two factors negatively impact the likelihood for success, as well as concepts of what constitutes success, and how it might be demonstrated.

In Inagaki's (2007: 41) review of 37 studies on the impact of C4D programs, only four provided any indication of long-term impacts, “and even among these studies impacts going beyond the immediate timeframe of the project are discussed through anecdotal accounts rather than systematic analyses”. Project implementation timeframes are usually too short to be able to assess long-term impacts. The average length of funding for projects reviewed by Inagaki (2007) was two years, and over half of the 37 studied had active project periods of one year or less.

Parks et al. (2005) suggest that assessing the impact of Communication for Social Change programs should look at short-term, intermediate and long-term impact. While Skuse (2006: 25) points out that understanding the behavioural impact of radio programs is “notoriously difficult and can only occur over the long-term”, he argues that “there is scope to set interim behaviour change indicators within ICD programmes that can and should be evaluated”. Souter (2008: 164) suggests that the best way of assessing “lasting and sustainable change” is to use longitudinal studies “undertaken some time (six months, two years, five years) after project closure”. However, he notes that the reluctance of donors to fund such studies is a particular problem in areas like ICD “where there is no strongly established evidence base of past experience on which to build” (Souter 2008: 164).

Strategies to Overcome the Challenges and Key Trends in C4D Evaluation

The following new conceptualisations of evaluation and shifts in evaluation practice have significant implications for understanding and evaluating C4D:

Evaluation is seen as an ongoing learning and organisational improvement process.

There is a shift from proving impacts to developing and improving initiatives.

The use of evaluative processes to support the development of innovations.

A shift from external to internal and community accountability (Lennie and Tacchi 2013).

These shifts respond in significant ways to the challenges and issues outlined above, and help to provide an environment for C4D evaluation practices that is ultimately supportive of better development planning and practice at all levels. For example, shifting to a greater focus on improvement highlights the value of focussing on progress towards long-term social change and the contribution made by C4D, as opposed to attempts to measure and attribute impact. Other key strategies that can help to overcome the challenges and issues identified above and achieve more sustainable C4D outcomes are outlined below.

Highlight the Value of Creative and Innovative Approaches to Evaluating C4D

A key finding from our UN consultations was that more openness, freedom and flexibility is needed in the selection and use of various evaluation approaches, methodologies and methods to ensure that they are appropriate and fit the aims of the C4D initiative. Innovative and creative participatory approaches, such as developmental evaluation (Patton 2011), ethnographic action research (EAR) (Tacchi et al. 2007) and the Most Significant Change technique (Davies and Dart 2005), can foster new understandings of local issues, facilitate community engagement and dialogue and personal and community change. We consider these approaches highly appropriate and effective for the evaluation of C4D. Creative processes such as digital storytelling, drawing pictures and maps, and photovoice techniques are also valuable. They are increasingly used in different stages of development research and evaluation as important elements of ethnographic and participatory action research methodologies (Liamputtong 2007; Rattine-Flaherty and Singhal 2009).

The feminist Nicaraguan C4D organisation Puntos de Encuentro (Lacayo 2006), which used flexible and creative methods to implement and evaluate its programs, is a good example of innovation, as is developmental evaluation, EAR and the participatory M&E approach used in AC4SC. All of these examples used an innovative, mixed methods, participatory approach to research and evaluation.

Use a Rigorous Mixed Methods Evaluation Approach

Systems and complexity theories highlight the need for methodological pluralism. Midgley (2006) notes that this is important to developing a flexible and responsive evaluation approach, which is essential in the evaluation of C4D interventions. Our UN consultations found that 80% of UN respondents and 79% of Expert Panel respondents considered a mixed methods approach “very important” in their work. A pragmatic, mixed methods approach can provide a fuller and more realistic picture of social change, shed light on different issues, and increase the strength and rigour of evaluation findings. It can capture different perspectives, is suitable for exploring complex situations and problems, can help to provide detail about local contexts, and can enable the collection of sensitive information and the inclusion of hard to reach groups (Bamberger et al. 2010). A mixed methods approach allows us to select from a broad range of methodologies and methods, providing exactly the kind of flexibility that is needed in the evaluation of C4D.

While many development agencies have used mixed method evaluations for several years, they have taken a somewhat ad hoc approach, and resources have usually not been available to increase their rigour (Bamberger et al. 2010). A participatory, mixed methods approach also requires a wider range of skills and knowledge to use effectively than standard evaluation approaches. This highlights the need to improve capacities and resources to more effectively undertake mixed methods evaluations of C4D.

Implement Long-Term, Holistic Evaluation Capacity Development at All Levels

There are many benefits in strengthening capacities in evaluating C4D among staff and stakeholders at all levels. A holistic approach to evaluation capacity development can increase the sustainability of C4D organisations and initiatives. This is a long-term approach that focuses on the development of organisations as a whole, rather than individual staff members (Horton et al. 2003). It requires a shift in how both evaluation and capacity development are approached and understood. The aim here is to develop organisations that continuously learn from success and mistakes, improve their practices, respond effectively to complex and rapidly changing contexts, and incorporate local innovation and ideas into the process (Hay 2010; Horton et al. 2003).

Institutionalising evaluation, developing an evaluation culture within organisations at all levels, and building the evaluation capacities of staff and stakeholders improves the quality of evaluation, understanding about evaluation and its role in the learning process, and C4D design and outcomes. However, as Pearson (2011) found in her work on a long-term capacity development project in Cambodia, there are many challenges and points of resistance that need to be identified and overcome. Strategies for overcoming these challenges include empowering local staff and communities involved in development projects.

Implement our Recently-developed Framework for Evaluating C4D

Another key strategy for addressing the challenges and issues of evaluating C4D is to implement our framework for evaluating C4D (Lennie and Tacchi 2013; Tacchi and Lennie 2014). Given its open, flexible and pluralistic approach, it can help to bridge the divide between upward accountability and learning-based approaches to the evaluation of development initiatives (Lennie and Tacchi 2014).

Our framework aims to assert and demonstrate the value, rigour and appropriateness of alternative approaches to evaluation. It is based on concepts and principles derived from systems and complexity theory, action research, feminist and gender-sensitive evaluation methodologies, new approaches to social change, and holistic approaches to community development, organisational change, and evaluation capacity development. These approaches promote ongoing learning from and continuous listening to a broad diversity of participants and stakeholders.

This framework proposes ways of critically thinking about the evaluation of C4D and suggests how to go about it. It consists of seven key, inter-related components – participatory, holistic, complex, critical, emergent, realistic and learning-based and principles that inform each component (see Figure 1).

Figure 1

Key Concepts in the Framework for Evaluating C4D

The framework fits most comfortably within a holistic approach to development based on systems and complexity thinking, which is increasingly seen as important to development and its evaluation (Miskelly et al. 2009; Ramalingam et al. 2008). It takes a participatory, flexible, mixed methods approach to research and evaluation, and incorporates action learning and a critical, realistic, approach to social change and evaluation. It advocates paying attention to power relations, difference (such as gender, age, ethnicity and literacy levels) and social and cultural norms in the process of researching and evaluating C4D. It emphasises people, relationships, processes, and principles such as inclusion, open communication, trust and continuous learning. This approach can help to reinforce the case for effective two-way communication and dialogue as central and vital components of participatory forms of development and evaluation that seek positive social change.

Implications for C4D Policy

The challenges and issues outlined above have a number of significant policy implications for C4D and communication for social change. They include policies related to: use of a broader range of evaluation approaches; providing sufficient time and resources for evaluations and evaluation capacity development; creating evaluation cultures within development organisations; and new understandings of accountability.

More openness, freedom and flexibility is needed in the selection and use of different evaluation approaches, methodologies and methods to ensure that they are appropriate and match the particular aims of the C4D initiative (Byrne and Vincent 2011). This requires a more open-minded approach to evaluation that draws on participatory and innovative methods that are more suited to the evaluation of C4D and can increase community participation, inclusion and empowerment. This process involves considering the strengths and limitations of all evaluation approaches, methodologies and methods, including participatory approaches. Addressing the challenges and issues we have identified also requires that those implementing C4D initiatives are given sufficient budgets and time for evaluation, including for longitudinal studies that identify expected, unexpected, positive and negative outcomes.

There is a clear need for more resources and support for evaluation capacity development at all levels, from grassroots to management. This requires a holistic approach that aims to develop learning organisations that continually improve their M&E systems and capacities, and can contribute to developing effective policies, strategies and initiatives that better address complex development goals. While the leadership of senior management is important to fostering organisational change towards an evaluation culture, Raeside (2011: 101) stresses the importance of staff recognising their own power to create change and the need to empower staff to act on the knowledge they gain from regularly interacting with communities. Raeside (2011: 101) argues that “If these staff are not empowered to act on this knowledge, it is unlikely that real power transformation will occur at this level, or that this information will ever trickle into mainstream development debates”. This suggests that organisations need to empower local M&E and C4D staff to act on the knowledge, insights and feedback obtained from their regular interactions with people at the community level, including those who could be important catalysts for social change.

We have noted the recent shift from evaluations being mainly based on upwards, external accountability to donors, to a greater stress on internal, personal and downwards, community-level accountability. David and Mancini (2011: 245) observe that over the last decade there has been an increased focus on accountability to primary stakeholders, accompanied by experimentation with “participatory approaches that address issues of power, justice and rights and open up new frontiers of enquiry, learning and understanding of change”. Likewise, Jones (2011: ix) comments on the emergence of innovative systems for feedback and increasing emphasis on transparency and accountability in development interventions. These new understandings of accountability have significant implications for evaluation reporting policies and practices in this field, including the establishment of effective two-way communication and feedback systems that can increase the success of participatory evaluations.

Conclusion

This paper has highlighted significant tensions, challenges and issues related to the effective and rigorous evaluation of C4D. We highlighted growing tensions between dominant results-based (upward accountability) approaches and emerging learning-based (improvement and effectiveness) approaches to evaluating development interventions. Our research has identified numerous contextual, structural and institutional challenges, issues and barriers, including problems with communication, attitudes towards C4D and evaluation, and with conceptualising, managing and planning the evaluation of C4D. We found a lack of skills and capacities in both evaluation and C4D, and a lack of appreciation, funding and support for alternative evaluation approaches that are more appropriate for the evaluation of C4D, compared with dominant RBM approaches. There are many challenges in assessing the impacts and outcomes of C4D, given the complexity of social change, difficulties with attribution and the unrealistic demands, targets and timeframes that are often imposed by donors.

We proposed various strategies that can help to address these challenges and issues, including highlighting the value of creative and innovative approaches to evaluating C4D. This can be achieved through examples such as AC4SC and Puntos de Encuentro, which both applied a participatory, mixed methods, learning-based approach to program development and evaluation. This approach can greatly strengthen the rigour of evaluation findings, provide a fuller and more realistic picture of social change, and provides exactly the type of openness, freedom and flexibility that is needed for the effective evaluation of C4D. We also argued that a holistic approach to evaluation capacity development can increase the sustainability of C4D organisations and initiatives. This involves seeing evaluation as a means of encouraging continuous learning, evaluative thinking and a culture of evaluation within organisations and communities, as well as a means of accountability.

A further suggestion was to implement our new framework for evaluating C4D, which demonstrates the rigour and cost-effectiveness (in the long run) of alternative evaluation approaches. We see the various approaches, theories and principles in our framework as vital for sustainable development that takes gender, power relations and social norms and the complexity of social change and evaluating C4D into account. However, it is important to take a critical, long-term view of the value of alternative evaluation approaches, given that they can be challenging to use sensitively and effectively. They also require the support of senior management and adequate time and resources to use in ways that are both rigorous and empowering for a broad diversity of people.

All of this has significant policy implications, including the need for more time, resources and support for long-term evaluation and evaluation capacity development, and more focus on internal, personal and downwards accountability though continuous feedback loops. Local staff also need to be empowered to act on the knowledge they gain from regularly interacting with communities so that real transformation and sustainable change can happen.

Figure 1

Key Concepts in the Framework for Evaluating C4D
Key Concepts in the Framework for Evaluating C4D

Tensions between Dominant and Alternative Approaches to Assessing the Outcomes of C4D

Dominant approaches Alternative approaches Tensions and issues
Dominance of instrumental, upward accountability-based approaches that focus on proving impacts, using linear cause-effect logic and formal reporting of results. Alternative approaches are not adequately resourced or supported and are often critiqued for lacking ‘objectivity’, ‘rigour’ and ‘validity’. Flexible, holistic interdisciplinary approach based on ongoing learning, improvement and understanding. Takes the complexity of social change and the particular context into account and focuses on outcomes that an initiative can realistically influence. Demonstrating the impact of C4D is complex and difficult. Dominant approaches discourage ownership of the evaluation process and learning from evaluation. Results are often biased towards positive outcomes, failures are not captured or learned from, and evaluations are not independent from donor influences.
Pressure to produce short-term results within rigid and unrealistic timeframes. This results in a focus on more tangible, short-term changes that are not good indicators of long-term social change. Seen as more important to focus on progress towards long-term social change and the contribution of C4D. This is a more realistic measure of effectiveness and provides practical recommendations for the implementation of policies and initiatives. Longitudinal studies are required but they are costly and one of the most difficult challenges in evaluation. Donors are reluctant to fund them. This means that there is a lack of strong evidence on which to build C4D research, which fuels scepticism.

Armytage, L. (2011) ‘Evaluating Aid: An Adolescent Domain of Practice’, Evaluation 17(3): 261–276. ArmytageL. 2011 ‘Evaluating Aid: An Adolescent Domain of Practice’ Evaluation 17 3 261 276 10.1177/1356389011410518 Search in Google Scholar

Balit, S. (2010a) ‘Communicating With Decision Makers’, Glocal Times 14. http://webzone.k3.mah.se/projects/gt2/viewarticle.aspx?articleID=181&issueID=21 BalitS. 2010a ‘Communicating With Decision Makers’ Glocal Times 14 http://webzone.k3.mah.se/projects/gt2/viewarticle.aspx?articleID=181&issueID=21 Search in Google Scholar

Balit, S. (2010b) ‘Communicating With Decision Makers (continued)’ Glocal Times 14. http://webzone.k3.mah.se/projects/gt2/viewarticle.aspx?articleID=183&issueID=21 BalitS. 2010b ‘Communicating With Decision Makers (continued)’ Glocal Times 14 http://webzone.k3.mah.se/projects/gt2/viewarticle.aspx?articleID=183&issueID=21 Search in Google Scholar

Bamberger, M., Rao, V. and Woolcock, M. (2010) Using Mixed Methods in Monitoring and Evaluation: Experiences from International Development. Manchester: The World Bank Development Research Group. BambergerM. RaoV. WoolcockM. 2010 Using Mixed Methods in Monitoring and Evaluation: Experiences from International Development Manchester The World Bank Development Research Group 10.1596/1813-9450-5245 Search in Google Scholar

Burns, D. (2007) Systemic Action Research: A Strategy for Whole System Change. Bristol: The Policy Press. BurnsD. 2007 Systemic Action Research: A Strategy for Whole System Change Bristol The Policy Press 10.2307/j.ctt9qgvn5 Search in Google Scholar

Byrne, A. (2008) ‘Evaluating Social Change and Communication for Social Change: New Perspectives’, MAZI 17. ByrneA. 2008 ‘Evaluating Social Change and Communication for Social Change: New Perspectives’ MAZI 17 Search in Google Scholar

Byrne, A. and Vincent, R. (2011) Evaluating Social Change Communication for HIV/AIDS: New Directions. Communication for Social Change Consortium for UNAIDS, Geneva. ByrneA. VincentR. 2011 Evaluating Social Change Communication for HIV/AIDS: New Directions Communication for Social Change Consortium for UNAIDS Geneva 10.1002/9781118241868.ch14 Search in Google Scholar

Conlin, S. and Stirrat, R. (2008) ‘Current Challenges in Development Evaluation’, Evaluation 14(2): 193–208. ConlinS. StirratR. 2008 ‘Current Challenges in Development Evaluation’ Evaluation 14 2 193 208 10.1177/1356389007087539 Search in Google Scholar

David, R. and Mancini, A. (2011) ‘Participation, learning and accountability: The role of the activist academic’, in Cornwall, A. and Scoones, I. (eds.) Revolutionizing Development: Reflections on the Work of Robert Chambers. London: Earthscan. DavidR. ManciniA. 2011 ‘Participation, learning and accountability: The role of the activist academic’ in CornwallA. ScoonesI. (eds.) Revolutionizing Development: Reflections on the Work of Robert Chambers London Earthscan 10.4324/9781003298632-35 Search in Google Scholar

Davies, R. and Dart, J. (2005) The ‘Most Significant Change’ (MSC) Technique. A Guide to Its Use. http://www.mande.co.uk/docs/MSCGuide.pdf DaviesR. DartJ. 2005 The ‘Most Significant Change’ (MSC) Technique. A Guide to Its Use http://www.mande.co.uk/docs/MSCGuide.pdf Search in Google Scholar

Earl, S., Carden, F. and Smutylo, T. (2001) Outcome Mapping: Building Learning and Reflection into Development Programs. Ottawa: IDRC. EarlS. CardenF. SmutyloT. 2001 Outcome Mapping: Building Learning and Reflection into Development Programs Ottawa IDRC Search in Google Scholar

Feek, W. and Morry, C. (2009) Fitting the Glass Slipper! Institutionalising Communication for Development Within the UN. A Discussion Document. 11th UN Inter-Agency Round Table on Communication for Development, Washington DC, 11–13 March, 2009. FeekW. MorryC. 2009 Fitting the Glass Slipper! Institutionalising Communication for Development Within the UN. A Discussion Document 11th UN Inter-Agency Round Table on Communication for Development Washington DC 11–13 March, 2009 Search in Google Scholar

Hay, K. (2010) ‘Evaluation Field Building in South Asia’, American Journal of Evaluation 31(2): 222–231. HayK. 2010 ‘Evaluation Field Building in South Asia’ American Journal of Evaluation 31 2 222 231 10.1177/1098214010366175 Search in Google Scholar

Horton, D., Alexaki, A., Bennett-Lartey, S. et al. (2003) Evaluating Capacity Development: Experiences from Research and Development Organizations Around the World. The Hague: International Service for National Agricultural Research. HortonD. AlexakiA. Bennett-LarteyS. 2003 Evaluating Capacity Development: Experiences from Research and Development Organizations Around the World The Hague International Service for National Agricultural Research Search in Google Scholar

Inagaki, N. (2007) ‘Communicating the Impact of Communication for Development. Recent Trends in Empirical Research’, Working Paper Series No. 120, Washington DC: World Bank. InagakiN. 2007 ‘Communicating the Impact of Communication for Development. Recent Trends in Empirical Research’ Working Paper Series No. 120, Washington DC World Bank 10.1596/978-0-8213-7167-1 Search in Google Scholar

Jallov, B. (2012) Empowerment Radio. Voices Building a Community. Gudhjem: Empowerhouse. JallovB. 2012 Empowerment Radio. Voices Building a Community Gudhjem Empowerhouse Search in Google Scholar

Jones, H. (2011) ‘Taking Responsibility for Complexity: How Implementation can Achieve Results in the Face of Complex Problems’, ODI Working Papers 330, June 2011, London: Overseas Development Institute. JonesH. 2011 ‘Taking Responsibility for Complexity: How Implementation can Achieve Results in the Face of Complex Problems’ ODI Working Papers 330 June 2011 London Overseas Development Institute Search in Google Scholar

Lacayo, V. (2006) Approaching Social Change as a Complex Problem in a World That Treats It as a Complicated One. The Case of Puntos De Encuentro. Nicaragua. Master of Arts thesis (Communication and Development), Ohio University, Athens, Ohio. LacayoV. 2006 Approaching Social Change as a Complex Problem in a World That Treats It as a Complicated One The Case of Puntos De Encuentro. Nicaragua. Master of Arts thesis (Communication and Development), Ohio University Athens, Ohio Search in Google Scholar

Leeuw, F. and Vaessen, J. (2009) Impact Evaluations and Development. NONIE Guidance on Impact Evaluation. Washington DC: The Network of Networks on Impact Evaluation. LeeuwF. VaessenJ. 2009 Impact Evaluations and Development. NONIE Guidance on Impact Evaluation Washington DC The Network of Networks on Impact Evaluation Search in Google Scholar

Lennie, J. and Tacchi, J. (2011) Researching, Monitoring and Evaluating Communication for Development: Trends, Challenges and Approaches. Report on a Literature Review and Consultations with Expert Reference Group and UN Focal Points on C4D. Prepared for the United Nations Inter-Agency Group on Communication for Development. New York: UNICEF. http://www.unicef.org/cbsc/files/RME-RP-Evaluating_C4D_Trends_Challenges__Approaches_Final-2011.pdf LennieJ. TacchiJ. 2011 Researching, Monitoring and Evaluating Communication for Development: Trends, Challenges and Approaches Report on a Literature Review and Consultations with Expert Reference Group and UN Focal Points on C4D. Prepared for the United Nations Inter-Agency Group on Communication for Development. New York: UNICEF. http://www.unicef.org/cbsc/files/RME-RP-Evaluating_C4D_Trends_Challenges__Approaches_Final-2011.pdf Search in Google Scholar

Lennie, J. and Tacchi, J. (2013) Evaluating Communication for Development: A Framework for Social Change. Abingdon: Routledge. LennieJ. TacchiJ. 2013 Evaluating Communication for Development: A Framework for Social Change Abingdon Routledge 10.4324/9780203078495 Search in Google Scholar

Lennie, J. and Tacchi, J. (2014) ‘Bridging the Divide Between Upward Accountability and Learning-Based Approaches to Development Evaluation: Strategies for an Enabling Environment’. Evaluation Journal of Australasia, 14 (1), 12–23. LennieJ. TacchiJ. 2014 ‘Bridging the Divide Between Upward Accountability and Learning-Based Approaches to Development Evaluation: Strategies for an Enabling Environment’ Evaluation Journal of Australasia 14 1 12 23 10.1177/1035719X1401400103 Search in Google Scholar

Lennie, J., Tacchi, J., Koirala, B. et al. (2011) Equal Access Participatory Monitoring and Evaluation Toolkit. Queensland University of Technology, University of Adelaide, and Equal Access Nepal. http://betterevaluation.org/toolkits/equal_access_participatory_monitoring. LennieJ. TacchiJ. KoiralaB. 2011 Equal Access Participatory Monitoring and Evaluation Toolkit Queensland University of Technology, University of Adelaide, and Equal Access Nepal. http://betterevaluation.org/toolkits/equal_access_participatory_monitoring. Search in Google Scholar

Lennie, J., Tacchi, J. and Wilmore, M. (2012) ‘Meta-Evaluation to Improve Learning, Evaluation Capacity Development and Sustainability: Findings from a Participatory Evaluation Project in Nepal’, South Asian Journal of Evaluation in Practice 1(1): 13–28. LennieJ. TacchiJ. WilmoreM. 2012 ‘Meta-Evaluation to Improve Learning, Evaluation Capacity Development and Sustainability: Findings from a Participatory Evaluation Project in Nepal’ South Asian Journal of Evaluation in Practice 1 1 13 28 Search in Google Scholar

Liamputtong, P. (2007) Researching the Vulnerable. A Guide to Sensitive Research Methods. London: Sage. LiamputtongP. 2007 Researching the Vulnerable. A Guide to Sensitive Research Methods London Sage 10.4135/9781849209861 Search in Google Scholar

Midgley, G. (2006) ‘Systems thinking for evaluation’, in Williams, B. and Imam, I. (eds.) Systems Concepts in Evaluation: An Expert Anthology. American Evaluation Association. MidgleyG. 2006 ‘Systems thinking for evaluation’ in WilliamsB. ImamI. (eds.) Systems Concepts in Evaluation: An Expert Anthology American Evaluation Association Search in Google Scholar

Miskelly, C., Hoban, A. and Vincent, R. (2009) How Can Complexity Theory Contribute to More Effective Development and Aid Evaluation? Dialogue at the Diana, Princess of Wales Memorial Fund. London: Panos. MiskellyC. HobanA. VincentR. 2009 How Can Complexity Theory Contribute to More Effective Development and Aid Evaluation? Dialogue at the Diana, Princess of Wales Memorial Fund London Panos Search in Google Scholar

Parks, W., Gray-Felder, D., Hunt, J. and Byrne, A. (2005) Who Measures Change: An Introduction to Participatory Monitoring and Evaluation of Communication for Social Change. New Jersey: Communication for Social Change Consortium. ParksW. Gray-FelderD. HuntJ. ByrneA. 2005 Who Measures Change: An Introduction to Participatory Monitoring and Evaluation of Communication for Social Change New Jersey Communication for Social Change Consortium Search in Google Scholar

Patton, M.Q. (2011) Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York: Guilford Press. PattonM.Q. 2011 Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use New York Guilford Press Search in Google Scholar

Pearson, J. (2011) Creative Capacity Development: Learning to Adapt in Development Practice. Sterling VA: Kumarian Press. PearsonJ. 2011 Creative Capacity Development: Learning to Adapt in Development Practice Sterling VA Kumarian Press Search in Google Scholar

Puddephatt, A., Horsewell, R. and Menheneott, G. (2009) Discussion Paper on the Monitoring and Evaluation of UN-Assisted Communication for Development Programmes. Recommendations for Best Practice Methodologies and Indicators, 11th UN Inter-Agency Round Table on Communication for Development, Washington DC, 11–13 March, 2009. PuddephattA. HorsewellR. MenheneottG. 2009 Discussion Paper on the Monitoring and Evaluation of UN-Assisted Communication for Development Programmes Recommendations for Best Practice Methodologies and Indicators, 11th UN Inter-Agency Round Table on Communication for Development Washington DC 11–13 March, 2009 Search in Google Scholar

Quarry, W. and Ramirez, R. (2009) Communication for Another Development. London: Zed Books. QuarryW. RamirezR. 2009 Communication for Another Development London Zed Books 10.5040/9781350219274 Search in Google Scholar

Raeside, A. (2011) ‘Are INGOs Brave Enough to Become Learning Organisations?’, in Ashley, H., Kenton, N. and Milligan, N. (eds.) How Wide are the Ripples? From Local Participation to International Organisational Learning, Participatory Learning and Action 63: 97–102. RaesideA. 2011 ‘Are INGOs Brave Enough to Become Learning Organisations?’ in AshleyH. KentonN. MilliganN. (eds.) How Wide are the Ripples? From Local Participation to International Organisational Learning, Participatory Learning and Action 63 97 102 Search in Google Scholar

Ramalingam, B and Jones, H. with Reba, T. and Young, J. (2008) Exploring the Science of Complexity: Ideas and Implications for Development and Humanitarian Efforts, ODI Working Paper 2nd ed., London: Overseas Development Institute. RamalingamB JonesH. RebaT. YoungJ. 2008 Exploring the Science of Complexity: Ideas and Implications for Development and Humanitarian Efforts, ODI Working Paper 2nd ed. London Overseas Development Institute Search in Google Scholar

Rattine-Flaherty, E. and Singhal, A. (2009) ‘Analyzing Social-Change Practice in the Peruvian Amazon through a Feminist Reading of Participatory Communication Research’, Development in Practice 17(6): 726–736. Rattine-FlahertyE. SinghalA. 2009 ‘Analyzing Social-Change Practice in the Peruvian Amazon through a Feminist Reading of Participatory Communication Research’ Development in Practice 17 6 726 736 10.1080/09614520903026884 Search in Google Scholar

Servaes, J., Polk, E., Shi, S. et al. (2012) ‘Towards a Framework of Sustainability Indicators for “Communication for Development and Social Change” Projects’, The International Communication Gazette 74(2): 99–123. ServaesJ. PolkE. ShiS. 2012 ‘Towards a Framework of Sustainability Indicators for “Communication for Development and Social Change” Projects’ The International Communication Gazette 74 2 99 123 10.1177/1748048511432598 Search in Google Scholar

Skuse, A. (2006) Voices for Change: Strategic Radio Support for Achieving the Millennium Development Goal. London: DFID. SkuseA. 2006 Voices for Change: Strategic Radio Support for Achieving the Millennium Development Goal London DFID Search in Google Scholar

Souter, D. (2008) ‘Investigation 4: Impact Assessment’, in BCO Impact Assessment Study. The Final Report, Building Communication Opportunities Alliance. SouterD. 2008 ‘Investigation 4: Impact Assessment’ in BCO Impact Assessment Study The Final Report, Building Communication Opportunities Alliance. Search in Google Scholar

Stern, E., Stame, N., Mayne, J. et al. (2012) Broadening the Range of Designs and Methods for Impact Evaluation. DFID Working Paper 38. SternE. StameN. MayneJ. 2012 Broadening the Range of Designs and Methods for Impact Evaluation DFID Working Paper 38. 10.22163/fteval.2012.100 Search in Google Scholar

Tacchi, J., Fildes, J., Martin, K. et al. (2007) Ethnographic Action Research Training Handbook. http://ear.findingavoice.org/ TacchiJ. FildesJ. MartinK. 2007 Ethnographic Action Research Training Handbook http://ear.findingavoice.org/ Search in Google Scholar

Tacchi, J. and Lennie, J. (2014) ‘A Participatory Framework for Researching and Evaluating Communication for Development and Social Change’ In K.G. Wilkins, T. Tufte and R. Obregon (eds.) The Handbook on Development Communication and Social Change. Chichester, UK: Wiley Blackwell: 298–320. TacchiJ. LennieJ. 2014 ‘A Participatory Framework for Researching and Evaluating Communication for Development and Social Change’ In WilkinsK.G. TufteT. ObregonR. (eds.) The Handbook on Development Communication and Social Change Chichester, UK Wiley Blackwell 298 320 10.1002/9781118505328.ch18 Search in Google Scholar

Tacchi, J., Lennie, J. and Wilmore M. (2013) ‘Critical Reflections on the use of Participatory Methodologies to Build Evaluation Capacities in International Development Organisations’. In S. Goff (ed.) From Theory to Practice; Context in Praxis. Selected Papers from the 8th Action Learning, Action Research World Congress Australia 2010, Toowong, Queensland: Action Learning Action Research Association: 150–160. TacchiJ. LennieJ. WilmoreM. 2013 ‘Critical Reflections on the use of Participatory Methodologies to Build Evaluation Capacities in International Development Organisations’ In GoffS. (ed.) From Theory to Practice; Context in Praxis. Selected Papers from the 8th Action Learning, Action Research World Congress Australia 2010 Toowong, Queensland Action Learning Action Research Association 150 160 Search in Google Scholar

Artículos recomendados de Trend MD

Planifique su conferencia remota con Sciendo