Entries from November 2007

Flesch and SMOG readability tests

November 30, 2007 · 4 Comments

One of the earlier readability tests is the Reading Ease formula developed by Rudolph Flesch. Considered to be almost as accurate a measure as the the Dale-Chall formula that preceded it, the reading ease formula has been noted to be easier to use. To compute it, one only has to count words, syllables, and sentences. Then, sentence length and word length are summed to compute the score. A low score suggests an extremely difficult piece of material while a high score implies a very easy read.

For details on the original Flesch formula, read more …

In 1978, David Foulger presented a simplified reading ease formula which is a modification of the original Flesch procedure. Foulger’s revision aims to cut down on word counting time which should make computations easier.

SMOG Readability Test

SMOG (Simple Measure Of Gobbledygook) is another readability test that’s easy to calculate manually. To calculate
the SMOG reading grade level, begin with the entire written work that is being assessed, and follow these four steps:

1. Count off 10 consecutive sentences near the beginning, in the middle, and near the end of the text.

2. From this sample of 30 sentences, circle all of the words containing three or more syllables (polysyllabic), including repetitions of the same word, and total the number of words circled.3

3. Estimate the square root of the total number of polysyllabic words counted. This is done by finding the nearest perfect square, and taking its square root.

4. Finally, add a constant of three to the square root. This number gives the SMOG grade, or the reading grade level hat a person must have reached if he or she is to fully understand the text being assessed. A few additional guidelines will help to clarify these directions:

* A sentence is defined as a string of words punctuated with a period (.), an exclamation point (!) or a question mark (?).
* Hyphenated words are considered as one word.
* Numbers which are written out should also be considered, and if in numeric form in the text, they should be pronounced to determine if they are polysyllabic.
* Proper nouns, if polysyllabic, should be counted, too.
* Abbreviations should be read as unabbreviated to determine if they are polysyllabic.

Not all pamphlets, fact sheets, or other printed materials contain 30 sentences. To test a text that has fewer than 30 sentences:

1. Count all of the polysyllabic words in the text.
2. Count the number of sentences.
3. Find the average number of polysyllabic words per sentence as follows:
Average= Total # of polysyllabic words/Total # of sentences
4. Multiply that average by the number of sentences short of 30.
5. Add that figure on to the total number of polysyllabic words.
6. Find the square root and add the constant of 3.

Perhaps the quickest way to administer the SMOG grading test is by using the SMOG conversion table.

Read more …

Categories: Pretesting
Tagged: , ,

Readability measures for print materials

November 29, 2007 · 2 Comments

Readability tests like the cloze procedure, Fog index, Flesch formula, and Smog are procedures to determine the reading ease of a piece of written material. It is done before materials are pretested with the target audience. Print materials with lower reading levels are easier to understand and remember even for highly literate audiences.

Readability tests are largely based on number of words, number of sentences or even number of polysyllables. These tests teach us a simple rule in readability — that when writing a piece of material, if we use shorter words (versus words with 3 syllables or more) and shorter sentences, chances are, our writing will be easier to understand by the average reader. The heuristic then is to simplify. See, I just used a 3-syllable word (heuristic) to mean “simple rule”. Strange how people tend to prefer to use complicated words to simpler words although it limits understanding the material they have written.

The Fog index is one method of analyzing written material to see how easy it is to read and understand. The resulting number indicates the number of years of formal education that a person requires in order to easily understand the text on the first reading.

Another psychological tool for measuring the effectiveness of communication is the cloze procedure. When first introduced, it was regarded mainly as a “new and better” way of determining the readability of printed English prose.

Read more …

Categories: Pretesting
Tagged: , , ,

Survey research

November 25, 2007 · No Comments

A farmer being interviewed for a survey, West Java, Indonesia

Limitations of the survey approach

‘But questionnaire surveys often take more time and resources than estimated, enslave researchers, and generate misleading data and unread reports. Some bad questionnaire surveys make rural people appear ignorant when they are not.’

Robert Chambers, Rural Development: Putting the Last First, Chapter III

Besides conducting focus group discussions, one can also pretest prototype communication materials through the use of survey research. Surveys focus on people and their beliefs, opinions, attitudes, motivations, and behavior. The goal in survey research is to infer the characteristics of a given population from samples drawn from that population.

There has been a demand for surveys because of their usefulness in evaluating the effectiveness of projects and development interventions, trying out prototype communication materials, setting the research agenda, testing research hypotheses, and designing extension strategies. The popularity of surveys also stems from the fact that survey data can be subjected to statistical analysis, which generate quantitative indicators often desired by donors and practitioners. As Robert Chambers and other experts have pointed out, surveys can be costly, inefficient and superficial unless these are carefully planned and combined with in-depth and more sensitive techniques carried out by a multi-disciplinary team.

Read more …

Categories: Evaluation of communication materials · Needs assessment
Tagged: ,

FGD process

November 24, 2007 · 7 Comments

Focus group discussion with farmers in Can Tho province, Vietnam

To ensure a successful focus group discussion (FGD), some logistical arrangements are necessary. These include invitations, group composition, transportation, venue, seating arrangments, timing, name tags, recording, and refreshments. From numerous FGDs I have conducted in Southeast Asia, I learned a few key lessons that could spell success or failure. One memorable lesson had to do with asking participants to wear name tags to enable the moderator to encourage shy participants. While FGD participants in the Philippines can be requested to wear name tags without a fuss, I found that in other countries, such is not the case. Early on, our research partners simply had not prepared the materials for name tags and when I brought it up, they were reluctant and brushed it aside. I have gone on to conduct many more FGDs — perhaps close to 100 — but I have stopped asking for name tags. The lesson here is that cultural differences exist and a communication researcher has to be sensitive to these.

For some visuals on the focus group discussion process, view this …

Categories: Evaluation of communication materials
Tagged: ,

Focus group discussion

November 24, 2007 · 2 Comments

Group interview

Focus group discussion with farmers, Victoria, Laguna, Philippines

In pretesting, there are various ways to gather data - direct observation, interpersonal simulation, and asking questions. In “asking questions” you have these options - focus group discussion, survey or experiment. Focus group discussion (FGD) can be used to solicit people’s reactions to all sorts of development topics, ranging from problems and aspirations of children (a series of FGDs I did for PLAN International) to resource management (for a “Three Reductions, Three Gains” media campaign in the Mekong Delta) and environmental radio soap opera for rural Vietnam.

FGD is a fast way to learn from the intended audience. As defined, it is a rapid assessment, semi-structured data gathering method in which a purposively selected set of participants gather to discuss issues and concerns based on a list of key themes drawn up by the researcher. It is a widely used technique in advertising and marketing where companies use it to pilot test their prototype commercials and products before they are launched in the market.

Read more …

Categories: Evaluation of communication materials
Tagged: , ,

Factors affecting validity of pretests

November 20, 2007 · No Comments

Pretesting a survey questionnaire with a farmer, West Java, Indonesia

When can one say that the pretest results he/she obtained could most likely predict audience reactions to the final version of a communication material? Reed (1974) pointed out these four factors that could affect the validity of pretests - subjects sampled, setting, manipulation, and measurement. Consider these questions that often confront pretesters: Should a random sample of the potential audience be interviewed for a pretest? Or will persons different from the intended audience do? Should we conduct our pretest in the same setting as the one in which the audience will see the final production or in restricted undisturbed settings? Should we present the finished production or will a mock-up do? Do we measure the intended effect of the material or some criterion other than the intended effect?

Read more …

Categories: Evaluation of communication materials · Pretesting
Tagged: , ,

Pretesting prototype communication materials

November 16, 2007 · No Comments

Interviewing a farmer to pretest print materials, Central Java, Indonesia

What is pretesting?

Pretesting is a systematic method of obtaining and measuring the reactions of a representative group of the intended audience of the communication material. Through their reactions, alterations can be made to further ensure that the materials can effectively communicate the intended message. A pretest is done to predict the likely reactions of the intended audience when presented with the material. Posters, leaflets, handbooks, radio spots, radio dramas, audio CDs, video and web pages are some of the materials that can be pretested.

Why pretest?

Pretesting is essential in order to ensure that our time, money, and manpower are well spent. It is a cost-effective way to avert negative results in the mass production of communication materials. Pretesting can inform the production staff whether text, illustrations and color are correctly understood or interpreted by the intended audience.

Since differences in socio-economic status, education and cultural patterns influence the way audiences perceive and interpret communication materials, miscommunication can likely happen unless planners obtain feedback from the people to whom the materials are aimed at. Pretesting will also ensure that there is nothing offensive and culturally unacceptable in those materials.

Read more

Categories: Evaluation of communication materials
Tagged: , ,

Introductory readings

November 4, 2007 · 2 Comments

Hello DC 132 class. I have uploaded several introductory readings materials on evaluation and pretesting. Please read them all and we shall discuss them during our first class meeting after my return on Nov. 13.

We also need to sort out the class schedule. I understand that DC 132 has been scheduled at 1-2 p.m., the most rush and sleepy time of day. I would like you to think of an alternative class schedule and let’s confirm this when I get back.

I’d like each of you to send me comments to this post, like your feedback on the ease or difficulty in downloading the materials, etc. If you prefer to send me an email, you can get my email address from Anabelle or Dick Alviola.

Happy reading.

Monina Escalada

Categories: Evaluation of communication materials
Tagged: , ,

Hello DC 132

November 4, 2007 · 7 Comments

Welcome to DC 132 (2nd sem, 2007-08). I wish I were there to meet you on the first week of classes but I have to be in Nha Trang, Vietnam to give a talk at a review workshop. You must have heard about our World Bank-funded development marketplace (DM) 2005 environmental radio soap opera in the Mekong Delta. The project has completed broadcasting 105 episodes and the workshop will discuss the results of the pre-test-post-test survey analysis, lessons learned, and future work.

Updated course outline

I have updated our course outline to bring it up to par to the demands of our cyber world. As you will see in the outline, we shall learn to analyze online sources (like this blog) and audiences (readers of your own blogs — friendster, multiply, 360.yahoo, blogger, etc. ). I am excited to start this blog which will be our medium of communication from here on. Please download and print the course outline so you can review it carefully and begin reading the suggested references. If you can’t find the specific reference listed in the outline, please feel free to substitute it with an appropriate reference. To download the outline, please click here.

Subsequent materials that I will post here will be in PDF format (to safeguard my intellectual property rights as well as those of other sources’) and I hope you have no trouble getting this in your PCs or at our mini-library or digital lab in DDC. If you have problems, please seek the help of Dick Alviola, our ICT coordinator in DDC.

Write comments to posts

To relay a message to me, please use the comments feature — give me your name and the names of your other classmates. I’d like to know how many have registered for this course and hope to hear from you — any queries or suggestions to make our semester even more fruitful. Please encourage everyone enrolled in DC 132 to write a comment to this post.

As this blog will contain stuff for all the courses I teach, please take note of the titles of the specific course you are enrolled in and the dates when I posted them to keep track of the sequence.

Best wishes.

Monina Escalada

Categories: Evaluation of communication materials
Tagged: ,