Entries from February 2008

Monitoring & evaluation of IEC materials

February 27, 2008 · 4 Comments

One common thread that runs through the media campaigns and entertainment-education projects that my partners and I have done is systematic monitoring and evaluation (M&E) of communication materials, process and effects. I decided to write about this because there seems to be an interest in this area. Just last week, a colleague was invited to give a talk on this subject at a planning workshop for a tree seedling systems project. I discussed with her our monitoring and evaluation framework which, I suggested, could be adapted for her talk.

So if you are starting a new development project which aims to achieve impact, it is assumed that communication will play a central role. Dr. Nora Quebral, the development communication guru, has emphasized this in the early 70s. For a development project to succeed, communication must be considered at the project’s planning stage and not just considered as an afterthought. To determine if communication can help bring about changes in knowledge, attitude and practices, then a monitoring and evaluation framework has to be in place.

What is monitoring & evaluation of IEC materials

  • It is a systematic and objective collection of information about activities, characteristics and outcomes of IEC materials — a media campaign, radio drama series, leaflets, booklets, posters, demonstrations, farmer experiments, etc.
  • It is conducted to make judgments about the performance, effects and impact of a campaign, drama series or a mix of communication materials.
  • It is done to inform decisions about future IEC projects.

Why systematic M&E is needed?

  • To develop program content and set targets
  • To track progress, troubleshoot problems, perform mid-course action
  • To determine if objectives were achieved
  • To quantify effects and impact
  • To understand cause and effect relationships
  • To help improve future communication strategies

- audience segmentation
- content design – framing, positioning
- branding – “Three Reductions, Three Gains”, “No Early Spray”
- media mix
- on-the-ground support

 

M&E events in the project cycle

In a typical project cycle, M&E events can take place at various stages, from planning stage (needs assessment) until project completion (review workshop). Below you will find various M&E activities occurring at different project milestones:

1. Needs assessment — focus group discussions, scoping studies, key informant interview
2.Project design and pre-implementation — stakeholders’ workshop, baseline survey, pretesting
3. Project implementation — site visits, key informant interviews
4. Monitoring of ongoing project — management monitoring survey, field visits
5. Evaluation of completed project — review workshop, field day

Communication research methods

In monitoring and evaluation of IEC materials, an array of communication research methods can be used. The matrix below presents various methods and the purpose of each.

In future posts, I will discuss each method in detail and upload sample instruments that we have used for audience analysis, baseline survey, management monitoring and post-tst survey. Content analysis and pretesting have been covered in earlier posts. But if you’re in a hurry and need some advice on specific communication research methods, please post a comment and I will help, pro bono always.

Method

Purpose

1. Audience analysis

To characterize audience (demographics, communication environment) to develop content of materials, set campaign targets

2. Baseline survey

To assess knowledge, beliefs and behavior – to document current scenario

3. Pretesting of prototype materials

To determine appeal, understandability of materials (radio drama, campaign materials)

4. Management monitoring survey

To track implementation plans and make adjustments as needed

5. Content analysis

To analyze the content of audience feedback

6. Post-test survey

To determine whether the project has achieved its objectives

Categories: Monitoring & evaluation
Tagged: ,

How to analyze audience research data

February 15, 2008 · 2 Comments

In our two recent initiatives on Entertainment-Education — Farm IPM Radio supported by the Rockefeller Foundation and Environmental Radio Soap Opera funded by the World Bank Development Marketplace 2005 Award — we took great pains in analyzing the audience. Research-based information on target audience characteristics, needs and preferences was to used to enhance the values grid and design the drama content to ensure that the soap operas resonate with the audience.

In an earlier post, we talked about the what, why and how of audience research. Supposing you have already gathered your audience analysis data, the next step is to review each questionnaire to make sure that the respondent has not skipped any question in the instrument. Once the completed questionnaires have been edited, the data need to be analyzed. This analysis phase can be relatively simple – such as manually determining the % of respondents giving specific answers or listing the various ways in which farmers said they might utilize a new practice. Whether your survey is simple or complex, it is best that the data are encoded, processed and analyzed using a statistical package.

The first step in data analysis is to code the data. Coding is the term used to describe the translation of question responses and respondent information to specific categories for analysis. The first stage of coding information involves the construction of a code book. A code book is a set of rules used to classify observations of variables into values that are transformed into numbers.

Sample Codebook

Q No.

Column

Variable Name

Codes

A

ID

Enter actual number

1

B

District

1 = Vi Thuy (1- 304)

2= O Mon (305- 606)

3= Chau Thanh A (607-908)

2

C

Sex

1= Male

2= Female

3

D

Source

1= Bought from seed grower

2= From own farm

3= Obtained from other farmers

4= Government extension office

5= Other (specify)

Variable naming rules
1. Keep column or variable names short; statistical packages such as SPSS won’t read variable names longer than 8 characters; variable names can have a maximum of 8 characters.
2. Must begin with a letter.
3. Cannot include a period, blanks or certain characters (!, ?, `, *)
4. Must be unique - this means that not two variables must have the same name.
5. Cannot include these reserved words (ALL, NE, EQ, LE, LET, BY, OR, GT, AND, NOT, GE, WITH)
6. Use one word – alphabet and numbers – but no space in between characters.
7. Variable names are not case sensitive, i.e. can be written in upper or lower case

Data entry
1. Use numerical codes for all questionnaire data.
2. Enter numbers only in a spreadsheet.
3. Enter only 1 answer for each column.
4. Create another column for another answer.
5. Remember that each row or line of the table represents one respondent or case, and each column represents one variable

ID

Sex

Age

Area

Variety

1

1

23

1.2

1

2

1

26

0.3

3

3

2

35

0.5

1

4

1

54

2.1

5

5

2

63

0.2

4

6

2

49

1.7


Data checking

1. After the data have been entered, check the file for wild codes and extreme values.

ID

Sex

Age

Area

Variety

1

11

23

1.2

1

2

1

26

0.3

3

3

5

35

0.5

1

4

1

54

2.1

5

5

2

63

0.2

4

6

22

49

1.7

1

In the sample data above, 11, 5, and 22 are considered wild codes for sex which generally has only two codes: 1=male, 2=female.

2. Some statistical packages enable one to check the data for outliers.

Data analysis
1. Get descriptive statistics (mean, median, mode, and range) for each numeric variable. Compare answers with earlier surveys - if possible. Are means too high or too low because of errors or extreme values from certain respondents?

2.Get frequency distributions for all variables. Compare results for different items in the set.

3. Analyze several variables at a time by looking at a group of related variables.
- Run crosstabs for pairs of variables you have selected.
- For nominal and ordinal variables, run a chi-square test and check if the relationship is significant.

Levels of measurement
The statistical measures to use with a variable depends on which type of scale it was measured on: nominal, ordinal, interval, or ratio. Nominal variables are numbers or words that are used merely as labels such as sex, district, and province. Ordinal variables are numbers or words that can be rank-ordered such as rating scales. Interval scale are numbers that have equal and consistent intervals but no true zero point, such as number of insecticide sprays. Ratio scale are real numbers that have a true zero point such as age, height, weight, and yield.

References
List, Dennis. 2005. Know your audience: a practical guide to media research. Original Books, Wellington, New Zealand.

Nachmias, C. F. and D. Nachmias. 1996. Research methods in the social sciences. St. Martin’s Press, New York.

 

Categories: Monitoring & evaluation
Tagged: ,

Student’s content analysis exercise

February 13, 2008 · 5 Comments

Three weeks ago, my devcom students embarked on their content analysis exercise. To give you a flavor of my students’ initial attempt at content analysis, here’s one report:

Content analysis of 5 food websites
by Paulo-Angelo Sosmeña
BSDC-3
Introduction

Most people nowadays rely on the Internet for a variety of purposes: research,
entertainment, communication with others, etc. From the Internet we get a wide range of information on science, fashion, news and food, among others.

Filipinos are great food lovers. Not only do we eat lots of different kinds of food but we also cook them to satisfy our hungry stomachs or to impress our girlfriends. Gone are the days when food preparers rely solely on cookbooks that they borrow from friends or buy from the bookstores. Now, with just a click of the mouse, we can search for different food recipes on the Internet free of charge by downloading them and we can try them out in the kitchen afterwards.

There are many websites about food. Though they cover one major topic, food, websites differ in their features and content.

Objectives - This content analysis aims to: 1) identify the contents of 5 sample food websites, and 2) compare and contrast their features.

Procedure
1) From the Internet, five food websites were chosen, namely:
A - Joyofbaking.com
B - Allrecipes.com
C - Recipes.com
D - Cooksrecipes.com
E - Pastrywiz.com

2) Parameters were laid out to establish the scope of the content analysis: number of authors, real name or pseudonym, layout color, text and image, site copyright, clickable image, food focus, advertisements and availability of contact information.

Results
The table below reveals that 80% of the sites showed the names of the authors or team members. Some 40% have single authors while the rest is of multiple membership where members an submit their recipes, rate, and give their comments. An equal 40% each revealed their real names or pseudonyms.

Two sites have single female authors while the rest consisted of both male and female. Some 60% of the sites showed the photos of the authors and team members. Eighty percent of the sites are copyrighted; 60% combined both text and image. Another 80% have clickable images inking the reader to its recipe guide.

In terms of food focus, all of the 5 sites focuses on cakes and pastries, 60% on drinks and beverages, 60% on meat dishes, and 60% on soups and stews. Forty percent of the sites advertise baking tools and kitchen utensils while an equal number promote cookbooks.

Table 1. General features of food websites.

Parameters

Joyofbaking

Allrecipes

Recipes

Cooksrecipes

Pastrywiz

Layout color

brown & gray

yellow & orange

green

white & blue

blue

Text and image

combination text & image

combination text & image

too textual, less image

too textual, less image

combination text & image

Site copyright

yes

yes

no

yes

yes

Clickable image

yes

yes

yes

no

yes

Number of authors

single

multiple

multiple

single

single

Author/member image

available

available

not available

not available

available

Name type

real name

pseudonym

pseudonym

real name

real name

Gender

female

male & female

male & female

female

female

Availability of contact info

yes

yes

no

yes

yes

Categories: Evaluation of communication materials
Tagged:

When did you last read a book?

February 10, 2008 · 25 Comments

In Britain, 2008 is the National Year of Reading and it was launched by no less than British Prime Minister Gordon Brown on Wednesday. In the headline of her article, Denise Winterman of BBC News asks: “Do we still need to read books to be clever? She goes on to ask: “With so many other ways to get information these days, do we still need books?”

Why we read books

We read books to be informed, entertained, inspired, to expand our horizons, to pass the time, and to lull us to sleep. We read books because they offer escape from our daily routine and because they’re there.

When I was a child, I remember reading the books published by Ginn & Company lying around the house. When I was in high school, there were no malls and fast food and I had no money. My extra money could only afford the 10-centavo jeepney fare to the Thomas Jefferson Library on Sta. Mesa St. (now Ramon Magsaysay Blvd.) in Manila. So off I went to that library on my free time. In air conditioned comfort, I would lose myself among the book shelves browsing at the thousands of book titles. I must admit that going to the Thomas Jefferson Library to read its books was my only imaginary escape from my limited resources. But it did expand my horizons and desire to achieve. I went on to finish high school and college which changed my life. I was offered a job even before graduation day and my first pay was PHP 305 a month, a far cry from the 10 centavos in my pocket. Later, I got an East-West Center scholarship which was like icing on a cake that my college education had baked. Well, the rest is history. I have moved on and the library is no longer there. In 1970 it moved to G. Araneta Blvd., Quezon City; in 1980, it moved to Buendia Avenue Extension (now Senator Gil Puyat Avenue), Makati City. Today, the library is located on the ground floor of the Annex Building of the U.S. Embassy on Roxas Boulevard.

Do Filipinos read books?

In February 2007, Virgilio Almario, national artist for literature, delivered a speech with a catchy title, “Nagbabasa Ka Ba?” (Do you read?) at the Read or Die Convention. He spoke about an ancient disease of the modern Filipino society–the problem of education and the related puzzle of why more than 90 percent of the Filipino nation do not read. He said that no one will admit that they do not read, even if the truth is their reading is narrow in scope and more often than not limited to the daily broadsheet.

According to Almario, reading is cultured, it is taught and it is carved into the hearts and minds of young children, sown into their very personalities, so that they will love the book like a precious gem or find it as delicious as their favorite treat in McDo or Jollibee.

A parallel observation was found by the 2007 National Book Development Board (NBDB) Readership Survey. Reading has declined across all socioeconomic groups, except those in the AB class. Public school students now read fewer books, newspapers, magazines, and comics than they did in the 2003 Readership Survey. For private school respondents, there was a slight increase in those reading comics. The median number of books read was three which suggests that even if half the adult population of the Philippines have read three or more non-school books in the past year, the other half have read only at most three, or no books at all.

Do college students and teachers read?

While we were on the topic, how can we improve the quality of instructional materials, Efren Saz also shared with us the results of a little survey he did in his Sociology 11 class on the books that his students read. He was surprised to find that most students cited the dictionary. I hope that survey result does not reflect the overall pattern in universities. If college students don’t read other books, how can we expect them to have better writing skills, a wide vocabulary, and a broad world view? In my undergrad days in UP Diliman, the English and Humanities teachers nurtured in us the importance of being a complete person. So whether we read The Book of Job, The Old Man and the Sea or Catcher in the Rye or appreciate the tapestries in the Sistine Chapel (from slides) a link was always made to being a complete person. In my classes, my mantra is read, read, read. But do college students really read books outside their required text, the dictionary or the Bible?

In the same breath, do teachers read? If you do, when was the last time you read a book outside your discipline and finished it? So what books have you recently read? How do you choose the books that you read?

To start this informal survey, I texted my son what book he is currently reading. He texted back: Dance Dance Dance by Haruki Murakami which he “just saw in the bookstore and judged by its cover.” I have just read Mark Haddon’s The Curious Incident of the Dog in the Night-Time, Paulo Coelho’s The Devil and Miss Prym and Michael Crichton’s Next. Since I live far away from Fully Booked, A Different Bookstore and Powerbooks, I often rely on my son’s hand-me-downs. I’m sure my next read will be the Murakami book. Interesting how children can influence the reading fare of parents. In fact, it was my son who introduced me to the Coelho books and a lot more.

Why do I get to read more books? Because they’re in my son’s book shelves and I no longer watch that much TV. Like other leisure activities, book reading is often displaced by TV viewing. So if you’re not an avid fan of telenovelas and fantaseryes, chances are you will be drawn to book reading. If you’re a book reader, chances are you write well.

So, why do you read? Send in your comments.

Categories: General
Tagged: ,

The 2008 Global Development Marketplace Competition

February 5, 2008 · No Comments

CGIAR photo -A Development Marketplace visitor sticks his vote for the People's Choice Award by the booth of the "Environment Radio Soap Opera for Rural Vietnam" project


In our university’s project development committee, one of our tasks is to scan the environment for funding opportunities and relay the information to various units. A great funding opportunity presented itself when the World Bank Development Marketplace Team has announced on Jan. 22 that the 2008 Global Development Marketplace (DM2008) competition began its search for early-stage, innovative ideas with potential for high impact in promoting Sustainable Agriculture for Development.

Proposals are welcome from all innovators: civil society groups, foundations and development agencies in the country of implementation can submit their ideas. Other applicants – individuals, government entities and businesses – can also apply, but only if they work with a local partner.

DM2008 is specifically focused on the agricultural challenges related to:
· linking small-scale farmers to input and product markets;
· improving land access and tenure for poor farmers; and
· promoting the environmental services of agriculture in addressing climate change and biodiversity conservation.

A total of US$4 million in awards is available, with a maximum award size of US$200,000 per project. The call for proposals closes on March 21, 2008 (23:00 GMT time). Applications will undergo a rigorous review drawing on hundreds of development experts, who will narrow down the pool of finalists to about 100 finalists.

The DM will then bring the finalists to Washington DC for the Marketplace event on September 24-25, 2008. Finalists will participate in knowledge exchange workshops and will present their ideas to the public, World Bank staff and an international jury comprised of senior development specialists. Some 25-30 winners will be announced at the close of the Marketplace.

Proposals must be written in English and submitted through the DM website, www.developmentmarketplace.org. Competition guidelines and step-by-step instructions are available in Arabic, Chinese, English, French, Portuguese, Russian and Spanish.

Having been there and done that, having won the DM2005 award, here’s my advice to interested readers. The jurors will look for innovative, practical ideas that have a high potential for impact. As in most prestigious awards, the project proposal which aims to provide the greatest good for the greatest number stands a better chance of winning. Here, quantification is essential so you need to specify how many persons will benefit from your project. You may wax lyrical about measuring impact using a qualitative approach such as “most significant change” but in the end donors still prefer to work with numbers. So if you’re thinking that your village extension project that will benefit 200 residents might be good enough, think again. Surely the World Bank development marketplace prize is best given to a project that has a wider mileage.

Check out the application guidelines and download the form from the DM website. At the finals, your project poster must be compelling and you must be able to demonstrate to the jurors your confidence in your project’s impact. In the final proposal, you will also need to reflect the amount of matching funds or counterpart resources from partner organizations that will be involved in your project.

Categories: Development communication
Tagged: ,