Entries categorized as ‘Monitoring & evaluation’

Impact evaluation of a media campaign

April 11, 2008 · No Comments

Yesterday, I was at a validation forum in which results of research supported by public funds were presented. One of the projects which is relevant to Devcompage was about impact, rate and spatial dissemination of agricultural technologies. Effectiveness of communication strategies used was measured through perception data, categorized into highly effective, effective, and so on — while their assessment of communication strategies was a mere listing of diffusion methods used — meetings, agro-fairs, radio program and techno-demo. What the report terribly lacked were objective indicators of media effectiveness and a more informed analysis of the diffusion and uptake of agricultural technologies covered by the study. The ACIAR scoping study on policy linkages could very well provide the overall context that will help one understand diffusion and adoption of agricultural technologies in the Philippines.

The next day, I received a copy of the impact evaluation report on a media campaign in Vietnam. Undertaken by two IRRI staff –Dr. Debbie Templeton, former impact specialist, and Zenaida Huelgas, associate scientist — this impact evaluation report is a fine example of how the use of an assessment framework, objective indicators, and careful analysis could provide scientific rigor. The report titled, “Three Reductions, Three Gains (3R3G) Technology in South Vietnam: Searching for Evidence of Economic Impact” presented evidence of adoption of 3R3G in terms of lowering seed rates and the resulting changes in farm production/cost structure and farmer profits were measured.

What is impact evaluation?

The World Bank independent evaluation group defines impact evaluation as the systematic identification of the effects – positive or negative, intended or not – on individual households, institutions, and the environment caused by a given development activity such as a program or project. Alternatively, we can say that Impact evaluation is an approach which measures the outcomes of an intervention apart from other possible factors. It is intended to determine whether the program had the desired effects on individuals, households, and institutions and whether those effects can be attributed to program intervention. Impact evaluations can also ascertain unintended consequences, whether positive or negative, on beneficiaries.

External review team visit fertilizer omission plots demonstration, Sragen, Central Java

Impact evaluations can be costly Rigorous impact evaluations can be costly. The World Bank has estimated that the average cost of a rigorous impact evaluation can run up to US$200,000 considering the cost of consultants. But the World Bank Independent Evaluation Group (IEG) came up with a booklet that details how one can conduct quality impact evaluations under budget, money and time constraints. The IEG website is a gold mine of useful and reliable information on monitoring and evaluation which must be bookmarked by all Devcompage readers.

Categories: Monitoring & evaluation
Tagged: ,

Monitoring & evaluation of IEC materials

February 27, 2008 · 4 Comments

One common thread that runs through the media campaigns and entertainment-education projects that my partners and I have done is systematic monitoring and evaluation (M&E) of communication materials, process and effects. I decided to write about this because there seems to be an interest in this area. Just last week, a colleague was invited to give a talk on this subject at a planning workshop for a tree seedling systems project. I discussed with her our monitoring and evaluation framework which, I suggested, could be adapted for her talk.

So if you are starting a new development project which aims to achieve impact, it is assumed that communication will play a central role. Dr. Nora Quebral, the development communication guru, has emphasized this in the early 70s. For a development project to succeed, communication must be considered at the project’s planning stage and not just considered as an afterthought. To determine if communication can help bring about changes in knowledge, attitude and practices, then a monitoring and evaluation framework has to be in place.

What is monitoring & evaluation of IEC materials

  • It is a systematic and objective collection of information about activities, characteristics and outcomes of IEC materials — a media campaign, radio drama series, leaflets, booklets, posters, demonstrations, farmer experiments, etc.
  • It is conducted to make judgments about the performance, effects and impact of a campaign, drama series or a mix of communication materials.
  • It is done to inform decisions about future IEC projects.

Why systematic M&E is needed?

  • To develop program content and set targets
  • To track progress, troubleshoot problems, perform mid-course action
  • To determine if objectives were achieved
  • To quantify effects and impact
  • To understand cause and effect relationships
  • To help improve future communication strategies

- audience segmentation
- content design – framing, positioning
- branding – “Three Reductions, Three Gains”, “No Early Spray”
- media mix
- on-the-ground support

 

M&E events in the project cycle

In a typical project cycle, M&E events can take place at various stages, from planning stage (needs assessment) until project completion (review workshop). Below you will find various M&E activities occurring at different project milestones:

1. Needs assessment — focus group discussions, scoping studies, key informant interview
2.Project design and pre-implementation — stakeholders’ workshop, baseline survey, pretesting
3. Project implementation — site visits, key informant interviews
4. Monitoring of ongoing project — management monitoring survey, field visits
5. Evaluation of completed project — review workshop, field day

Communication research methods

In monitoring and evaluation of IEC materials, an array of communication research methods can be used. The matrix below presents various methods and the purpose of each.

In future posts, I will discuss each method in detail and upload sample instruments that we have used for audience analysis, baseline survey, management monitoring and post-tst survey. Content analysis and pretesting have been covered in earlier posts. But if you’re in a hurry and need some advice on specific communication research methods, please post a comment and I will help, pro bono always.

Method

Purpose

1. Audience analysis

To characterize audience (demographics, communication environment) to develop content of materials, set campaign targets

2. Baseline survey

To assess knowledge, beliefs and behavior – to document current scenario

3. Pretesting of prototype materials

To determine appeal, understandability of materials (radio drama, campaign materials)

4. Management monitoring survey

To track implementation plans and make adjustments as needed

5. Content analysis

To analyze the content of audience feedback

6. Post-test survey

To determine whether the project has achieved its objectives

Categories: Monitoring & evaluation
Tagged: ,

How to analyze audience research data

February 15, 2008 · 2 Comments

In our two recent initiatives on Entertainment-Education — Farm IPM Radio supported by the Rockefeller Foundation and Environmental Radio Soap Opera funded by the World Bank Development Marketplace 2005 Award — we took great pains in analyzing the audience. Research-based information on target audience characteristics, needs and preferences was to used to enhance the values grid and design the drama content to ensure that the soap operas resonate with the audience.

In an earlier post, we talked about the what, why and how of audience research. Supposing you have already gathered your audience analysis data, the next step is to review each questionnaire to make sure that the respondent has not skipped any question in the instrument. Once the completed questionnaires have been edited, the data need to be analyzed. This analysis phase can be relatively simple – such as manually determining the % of respondents giving specific answers or listing the various ways in which farmers said they might utilize a new practice. Whether your survey is simple or complex, it is best that the data are encoded, processed and analyzed using a statistical package.

The first step in data analysis is to code the data. Coding is the term used to describe the translation of question responses and respondent information to specific categories for analysis. The first stage of coding information involves the construction of a code book. A code book is a set of rules used to classify observations of variables into values that are transformed into numbers.

Sample Codebook

Q No.

Column

Variable Name

Codes

A

ID

Enter actual number

1

B

District

1 = Vi Thuy (1- 304)

2= O Mon (305- 606)

3= Chau Thanh A (607-908)

2

C

Sex

1= Male

2= Female

3

D

Source

1= Bought from seed grower

2= From own farm

3= Obtained from other farmers

4= Government extension office

5= Other (specify)

Variable naming rules
1. Keep column or variable names short; statistical packages such as SPSS won’t read variable names longer than 8 characters; variable names can have a maximum of 8 characters.
2. Must begin with a letter.
3. Cannot include a period, blanks or certain characters (!, ?, `, *)
4. Must be unique - this means that not two variables must have the same name.
5. Cannot include these reserved words (ALL, NE, EQ, LE, LET, BY, OR, GT, AND, NOT, GE, WITH)
6. Use one word – alphabet and numbers – but no space in between characters.
7. Variable names are not case sensitive, i.e. can be written in upper or lower case

Data entry
1. Use numerical codes for all questionnaire data.
2. Enter numbers only in a spreadsheet.
3. Enter only 1 answer for each column.
4. Create another column for another answer.
5. Remember that each row or line of the table represents one respondent or case, and each column represents one variable

ID

Sex

Age

Area

Variety

1

1

23

1.2

1

2

1

26

0.3

3

3

2

35

0.5

1

4

1

54

2.1

5

5

2

63

0.2

4

6

2

49

1.7


Data checking

1. After the data have been entered, check the file for wild codes and extreme values.

ID

Sex

Age

Area

Variety

1

11

23

1.2

1

2

1

26

0.3

3

3

5

35

0.5

1

4

1

54

2.1

5

5

2

63

0.2

4

6

22

49

1.7

1

In the sample data above, 11, 5, and 22 are considered wild codes for sex which generally has only two codes: 1=male, 2=female.

2. Some statistical packages enable one to check the data for outliers.

Data analysis
1. Get descriptive statistics (mean, median, mode, and range) for each numeric variable. Compare answers with earlier surveys - if possible. Are means too high or too low because of errors or extreme values from certain respondents?

2.Get frequency distributions for all variables. Compare results for different items in the set.

3. Analyze several variables at a time by looking at a group of related variables.
- Run crosstabs for pairs of variables you have selected.
- For nominal and ordinal variables, run a chi-square test and check if the relationship is significant.

Levels of measurement
The statistical measures to use with a variable depends on which type of scale it was measured on: nominal, ordinal, interval, or ratio. Nominal variables are numbers or words that are used merely as labels such as sex, district, and province. Ordinal variables are numbers or words that can be rank-ordered such as rating scales. Interval scale are numbers that have equal and consistent intervals but no true zero point, such as number of insecticide sprays. Ratio scale are real numbers that have a true zero point such as age, height, weight, and yield.

References
List, Dennis. 2005. Know your audience: a practical guide to media research. Original Books, Wellington, New Zealand.

Nachmias, C. F. and D. Nachmias. 1996. Research methods in the social sciences. St. Martin’s Press, New York.

 

Categories: Monitoring & evaluation
Tagged: ,