Discussing pretesting results, JInhua, Zhejiang, China

In Jinhua, Zhejiang province last week, 14 student enumerators pretested our draft baseline survey questionnaire with rice farmers. When the college students returned from the field, they spoke about farmers’ difficulty understanding the term “wild flowers” in the questionnaire.  Growing wild flowers is an ecological engineering method aimed to restore and enhance important ecosystem services to provide adequate crop health.

Tridax-procumbens-L. Photo credit: S. Villareal (IRRI)

Vernonia-cinerea-(L.)-Less. Photo credit: S. Villareal (IRRI)

The term “wild flowers” was used in a series of attitude/belief statements in our draft questionnaire. Here are some examples:

  1. Increasing wild flowers on the bunds is a waste of time.
  2. Increasing wild flowers on bunds is easy to do.
  3. It is difficult to increase wild flowers on bunds because nearby paddy fields use herbicide.
  4. We cannot increase the wild flowers on bunds because we burn our rice straw.
  5. Increasing wild flowers on bunds is additional burden to farmers.
  6. It is difficult to increase wild flowers on bunds because farmers will step on them.
  7. Our bunds are narrow so there is no place for wild flowers.
  8. I am willing to try increasing wild flowers in the bunds to learn more about what they can do

Students reported that farmers thought that wild flowers are useless and they should not waste time growing them and answering questions that had to do with these flowers.  This pretest feedback pointed to a lack of understanding between scientists and farmers on what the words meant. As used in the  questionnaire, wild flowers are plants which are not cultivated but produce nectar-rich flowers to attract natural enemies. Many are growing wildly around rice paddies, hills and roadsides. On the other hand, farmers thought that wild flowers are utterly useless plants.  After a long discussion on what term will best capture uncultivated flowers, it was agreed that the term, “beneficial flowers” be used instead.

What is questionnaire pretesting

The pretest is a try-out of the questionnaire to see how it works and whether changes are necessary before the start of the actual survey. About 15 to 20 respondents, whose characteristics are reasonably similar to the survey population, will be adequate for a pretest.  The questionnaire is then revised and finalized on the basis of pretest results.  Pretesting guidelines will facilitate the conduct of a pretest.

Why pretest a survey questionnaire

The pretest provides a means of catching and solving unforeseen problems in the use of the questionnaire, such as the phrasing and sequencing of questions.  Linguistic and cultural differences also complicate the task of  questionnaire development, making pretesting all the more indispensable. The pretest enables one to:

1) improve the wording of the questionnaire;
2) correct and improve translation of technical terms;
3) check the accuracy and adequacy of the questionnaire’s instructions such as “skip” and “go to”;
4) eliminate unnecessary questions and add necessary ones; and
5) estimate the time needed to conduct the interview.  In the illustration below, excerpts of pretest results are reported to illustrate how pretesting yielded constructive suggestions which served as the basis for improving a questionnaire.

Years ago, we conducted a rather thorough questionnaire pretest in Iloilo province, Philippines. Here’s  the pretest report of Danny N. Valenzuela, our research collaborator for a farmer survey on rice seed health.

Sample pretest results:  suggestions from the field

The survey questionnaire for the project “Farmers’ perceptions and attitudes toward seed health for crop management” was pretested in Pandac, Pavia, Iloilo. The goals of the pretest were to determine the reactions of rice farmers to the questionnaire, estimate the time needed to complete the interview, validate the translation of key technical terms used, find out whether the respondents could understand the technical terms,  and ascertain whether the sequence of the questions solicited the desired information. A survey enumerator was hired to assist in pretesting the instrument with 15 rice farmers.

The following observations and corresponding recommendations were made:

1)  In this village, the first and second crops are both grown during the wet season, so the reference to the first one as the wet season crop and the second one as the dry season crop does not apply.  Recommendation: Provide the necessary indicators for the interviewers to guide them if they are confronted with decision problems in this area.

2) The second cropping was not realized because of drought (question 4:  Last year for the second cropping, what was the area on which you direct-seeded and/or transplanted?    Some farmers did not plant rice and instead planted watermelon or tomatoes.  Recommendation:  Point this out to the interviewers during the orientation.

3) In question 5 (What cropping pattern did you practice?),  the farmers found the term “cropping patterns” difficult to comprehend.  The question cannot be properly answered without explaining the term or making it a leading question.  Recommendation:  Modify the question to instead ask what crop was grown in a particular season. The farmers   also mentioned a third cropping season during which they plant  crops” such as watermelon, tomatoes, and mungbean in lieu of rice.

4) Question 6 (Last cropping season, how much (in pesos) did you spend for rice seeds, insecticides, fungicides, herbicides, fertilizer, irrigation and labor  for pesticide application?) entailed a lot of time to answer because the interviewer often had to add up the figures given by farmers. Recommendation: Remind the interviewers to bring calculators with them.

20 Responses to What questionnaire pretesting reveals

  1. jen says:

    Are the participants in the pretesting the same people who will be in the survey?

  2. monina escalada says:

    Jen, no, the farmers interviewed in the questionnaire pretest won’t be included in the final sample. Based on the pretesting feedback, some of the questions will be revised. In our Jinhua pretest, the farmers didn’t like the term “wild flowers” and that has been changed to “beneficial flowers” so the meaning is slightly altered.

  3. jen says:

    we’re pretesting our questionnaire in a barangay adjacent to the one we target. i was wondering if the pretest should be done in the same barangay.

  4. monina escalada says:

    Jen, it’s not necessary to do your questionnaire pretest in the same study barangay. The rule is to pretest the questionnaire with respondents whose characteristics are reasonably similar to the ultimate respondents. Best to pretest in the barangay adjacent to your sample barangay so that your sampling population won’t be reduced.

  5. Sha says:

    Can these beneficial flowers adapt to Philippine climate if planted here?

  6. monina escalada says:

    Sha, the beneficial flowers in the photo are grown in the Philippines. The close-up shots must have made them look exotic. Our ecological engineering project aims to encourage rice farmers to plant these around their rice paddies to attract bees and other natural enemies.

  7. Dennis says:

    hi ma’am
    who are the survey enumerator?

  8. monina escalada says:

    Dennis, a survey enumerator is an interviewer, a person who interviews the respondents. In Jinhua, China, the interviewers are students of Jinhua College.

  9. Jed says:

    We, as DevCom practitioners, need to “bridge the gap” (yun). Sometimes, we get so familiar of our the “devcom subculture” when we’re together that we forget that we should be reaching out, like this. You’ve just written great examples, Ma’am.

    I just realized that it’s not that simple even in making questionnaires for farmers. We have to be sensitive of how people would give meaning to things (which might be different to how we do).

    Bringing this to the context of the student pub, as you always do in our classesm, Ma’am, how can the Amaranth do a pretesting of its semestral magazine? Editors probably are trapped in their own mini-culture that issues that students deem important are not quickly seen.

    Or perhaps, students don’t really care to voice out their concerns…How do we know?

  10. monina escalada says:

    Jed, for your student publication, The Amaranth, what you can do is to encourage raders to contact you via email, web, snail mail, and give their feedback (suggestions, comment) on the latest issue of The Amaranth.

    Alternatively, you can do a poll on what concerns or issues students want The Amaranth to tackle in the future.

  11. jona says:

    Ma’am, after pretesting the questionnaire and improving it will there be any other step (like another testing of materials) before choosing and conducting the final sample? Is pretesting only done once?

  12. pam says:

    Does pretesting a survey questionnaire always help us determine the strengths and weaknesses of our survey concerning question format, wording and order?

  13. monina escalada says:

    Jona, questionnaire pretesting is often done once. Pretest results are used to improve and finalize the questionnaire.

  14. monina escalada says:

    Pam, yes, questionnaire pretesting helps the researcher to determine whether the words and phrasing of the questions could be understood, and the format and sequence make sense to the respondents.

  15. Buen Josef says:

    Pretesting the survey questionnaire is really an essential step in social research, especially to students and social research practitioners alike in achieving genuine results. One of the benefits we can get out of pretesting the survey questionnaire is the comprehensibility of the translation of the contents into vernacular (cebuano,waray, etc.). As we often use English on our survey questionnaire/interview schedule, we sometimes forget the appropriate terms in the vernacular. We can see how useful it is if our respondents are people from the remote barrios who only know English words like Yes and No. To see whether the translations we did are correct and comprehensible to our respondents, we must employ pretesting our questionnaires first before actually doing the survey. With this, we can (at least) help bridge the language gap to our respondents.

    When I did my thesis, pretesting the interview schedule did help a lot since I used the local language (cebuano) and my respondents are rural and urban mothers. In my pretest, I found out that there are some areas where my respondents cannot understand (even my cebuano).

  16. Junette Dawn says:

    Indeed, pretesting survey questionnaire is essential, if not important, in any social research or studies. Not only does it save time and money, it also entails researchers to correct and improve their materials prior distribution. If one does not conduct a pretest, imagine what will happen to the study if flaws are found on the material after it has been administered or distributed. The consequences are horrifying.

    Identifying and addressing problems in surveys early in the process gives researchers more time to improve the instrument, whether it concerns on the validity, comprehensibility, etc. This stage in survey development has been overly emphasized because it ensures materials convey clear and effective messages to the audience. In the communication world, if these are not achieved, it is as if you did not communicated at all.

  17. Jovelyn says:

    I can certainly attest that a very essential part of the survey questionnaire construction process is pretesting. In pretesting, you will be able to determine that your research instrument have able convey the message to your target audiences what you intended and that they understand what they are supposed to do.
    Take for instance in my case where I just recently conducted a pretest on our survey questionnaire. If not for pretesting, we would have not able to check for glitches in wording of questions, lack of clarity of instructions or anything that could impede the instrument’s ability to collect data. So for you to determine the effectiveness of your survey questionnaire, it is necessary to pretest it before actually using it. Pretesting can help you determine the strengths and weaknesses of your survey.
    So to those who will be conducting a survey for your research, the temptation to rush over things, like using just a convenience sample, should be avoided.

  18. robel june says:

    Pretesting is said to be the critical step that happens in the early stage of survey development process, pretesting is used to determine and correct the errors of the instruments under development. In doing the proper pretesting, it involves a clear comprehension or understanding of each survey question’s target.

    Pretesting supports the researcher to illuminate the survey’s goals, which leads the complete research into a development. Imagine a survey without a pretest? When the instrument is administered, it is when you’ll determine the flaws or errors of your instrument in which you will encounter several problems.

    The importance of pretesting is that, researchers will have enough time to improve or correct the flaws of the instrument. Also, by pretesting, the information that you desire to communicate and transfer to your target audience will be properly delivered.

  19. Vinesse says:

    Most of the time, people have this common attitude of judging our works according to our own personal judgments and understanding. But, dependence on human instincts should not be applied in studies knowing that a single mistake could lead to great losses and impacts more than our imagination. So, for instance, how could a researcher make sure that a questionnaire is as clear and understandable when it reaches the target audience? By a try-out of how the material would work or “Pretesting”.

    Yes, I do agree that questionnaire pretesting is very important and should be a must in every study. It helps researchers in detecting possible problems (including vague and unclear questions, wrong sequencing of questions, unclear instructions and others) in the material and most importantly, it provides room for necessary changes and revisions. Since the results of the pretest will be the basis in finalizing the material, rest assured the material will be converted to a more effective one in achieving its needed information from the target audience. Pretesting prevents any waste of time and money in the side of the stakeholders and possible feelings of regret to the researchers also.

    So, if pretesting will not be conducted, misinterpretations will surely intervene in the process of conveying message to the target audience, therefore, defeating the main goal in the context of communication, which is to be understood.

  20. Naji Juntilla says:

    Pretesting is very important before conducting a study. This will help
    the researcher to determine errors and if the questionnaire is
    suitable for the target audience of the study. Revisions of the
    material are done after the pretesting to make it more effective and
    understandable. Prior to the study with the help of pretesting the
    researcher can detect flaws in the material that will be used.
    Conducting a pretesting means producing a material that is suitable,
    properly organized, understandable and even more appropriate to the
    target audience that will prevent misconception and confusion.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>