Dr. Carmen Thoennissen (SDC) and KL Heong (IRRI) visit "Three Reductions, Three Gains Project", Vietnam
External review of an integrated crop resource management project, Indonesia
To track change and demonstrate impact, indicators are needed and developed with various stakeholders. Among the stakeholders, there has to be a consensus on what an evaluation is for, whose objectives are being served, and how the data will be used. Communication evaluation is an approach, which measures the outcomes of an intervention apart from other possible factors. It is intended to determine whether the program achieved the desired effects on individuals, households, and institutions and whether those effects can be attributed to program intervention. Impact evaluations can also ascertain unintended consequences, whether positive or negative, on beneficiaries.
What is campaign impact assessment?
Campaign impact assessment examines effects at the community, state, national or international level, or a campaign’s long-term effects or identifies whether it is justified to say that the campaign caused any social or policy change. The evaluation may assess:
- The causal relationship between the campaign’s activities and its outcomes;
- Longer-term outcomes as a result of behavior or policy changes (e.g. lower cancer rates, less violence in schools); and
- Systems-level change, through policy change or the creation of new institutions, procedures or groups.
The use of experimental research designs enables one to make more definitive conclusions about the impact of a campaign. Such rigorous designs require random assignment of target audiences to “treatment” and “control” groups or conditions, and require evaluation or research expertise to implement it. It answers the question:
If x, then y
Or
If the campaign is implemented, then the outcome occurs
In situations where it is not possible to have a control group of participants who have not been exposed to the campaign, quasi-experimental designs are used. These do not require random assignment but require a comparison group.
Steps in communication evaluation
1. Review communication objectives
2. Formulate hypotheses
3. Select impact indicators
4. Develop and apply impact monitoring methods – interviews, focus group discussions, transect walks, observation
5. Assess impact
Impact planning framework
A useful tool that can be used to plan an impact assessment of an extension or communication initiative is this framework:
Project title: (e.g. Reducing farmers’ early season spray for leaffolder control)
Brief description of the technology:
Impact or change desired: (e.g., reduce early season insecticide sprays by 25%; change farmers’ beliefs in leaffolder management by 30%)
Target audience: (e.g., 20,000 farmer households in Tan Tru and Tan Thanh districts, Long An Province, Vietnam)
Variables to be monitored: (e.g., farmers’ insecticide use; farmers’ target pests’ crop stages when farmers spray’; insecticide sales; belief index)
Methods to be used for each variable: (e.g., farmer surveys; focus group interviews; participant observation; assessments of attitudes)
Julia Coffman (2003, June- Lessons in evaluating communications campaigns: Five case studies. Cambridge, MA: Harvard Family Research Project) examined the process of evaluating five different campaigns whose purposes ranged from individual behavior change to policy change, and showed how evaluators have addressed some evaluation challenges. Five case studies of campaign evaluation are presented which illuminate the real experiences of campaign evaluations. The paper also features cross-case lessons that highlight important findings and themes.
Starting tomorrow, 13 December 2007, a national congress for development communication students will take place at the University of the Philippines in Los Banos. The theme of this year’s congress is “Emphasizing the role of development communication for a sustainable environment.” I’ve been invited to give a talk at the opening program tomorrow afternoon on “Environmental radio soap for rural Vietnam,” the DM2005-supported project that recently won the COM+ prize.
