What is and why “pre-testing”?

Testing materials before they are used in live surveys allows to make certain that questionnaire is accurate, non-leading and reliable. This is often carried out at a late stage and using finished materials, i.e. the final version of a the questionnaire.

Considerable professional and financial investment is therefore at stake, and pre-tests can constitute some of the most difficult and contentious step within a household survey project. People often think that testing a survey takes a long time. They think they don’t have the time or resources for it, and so they end up just running the survey without any testing. This is a big mistake. Even testing with one person is better than no testing at all. So if you don’t have the time or resources to do everything in this guide, just do as much as you can with what you have available.

As a good practice, the entire questionnaire should be submitted to a number of procedures to ensure there are no questions likely to test anything but respondent real status or opinion. There are extensive guidelines on this subject, belwo is a summary of the main points to look at.

Objectives

The overall objectives and the focus of the pre-test, together with the results shoudl be properly documented before starting it.

  • Questions are clear i.e. respondents do not misinterpret the questions (questions are not ambiguous or difficult to understand);

  • Response categories are comprehensive and adequate for the assessed population; any answer falling into the “other (specify)” category of a multiple choice question and that constitutes about 5 percent or more of all answers to that question should be considered as a serious candidate for a separate answer category of its own. New codes for common answers that were not included in the original questionnaires will need to be created

  • Flow of questions within the questionnaires is adequate;

  • Difficult or sensitive questions/modules are identified so that extra training can focus on these questions during the fieldworker training;

  • Translations are accurate, ie. translated questionnaires are working correctly. Changes in wording or improved translation will need to be incorporated when required;

  • Interviewer instructions in the Instructions for Interviewers, as well respondent informed consent in questionnaire are clear and sufficient;

  • Average duration of interviews is calculated in order to plan the fieldwork and the daily workload per interviewer/team

Document the organisation of the Pre-test

  • Clusters/Number of interviews selected for pre-test: Describe the pre-test locations where were the households located for the pre-test, how and why were these locations selected, etc. Note that often, what is important during the pre-test is more the quality of the observation within each interview than the number of interviews itself.

  • Personnel: Present the trainers and interviewers (trainees) of the pre-test. Include information on the future involvement of the participants in the rest of the assessment process.

  • Training: List the dates and content of the pre-test training, as well as how it was organised. Some detail is useful on agenda and training methodology, as it can serve as lessons for the main training. Include other details as relevant: Venue, recommendations for main training, etc.

  • Fieldwork: Provide the dates of actual pre-test fieldwork. Also detail on organisation (logistics, teams, areas, etc.) is very useful.

  • Findings: Describe how the observations from the pre-test were collected and discussed and what process for making changes to the final questionnaires was used.

How many respondents?

As a general rule, one should aim to pretest a form with at least 5 to 10 respondents and 2 different interviewers. If the survey is more complex, as studied by reasearcher, a sample of 30 respondents is recommended.

Even with this small number of people, a surprisingly large number of improvements can be made. Try to get within those 5 to 10 respondents a range of different people who are representative of the target group for the questionnaire. Usually, most of respondents will have the same problems with the survey, so even with such small number of people, it should be possible to identify most of the major issues. Adding more people might identify some additional smaller issues, but it also makes pretesting more time consuming and costly.

How to observe pre-test?

Note that for the pre-test, it’s important to have 2 persons supporting the interview. One regular enumerator and an additional person to take note of each observation during the interview.

  • Respondents should be asked to complete the survey while “thinking out loud”. They should tell you exactly what comes into their mind so that the observer can take notes on everything they say. This which can include paraphrasing, providing retrospective thinking or providing judgments of their confidence in what each question means.

  • The observer should look for places where the respondent hesitate or make mistakes.

  • A debriefing can be organised after each pre-test interview to ensure that potential additional observations from the enumerator are also inclued in the observation notes.

Pre-test Results and Recommendations

Relating to the objectives listed above, this section should include findings from the actual data gathered as well as the qualitative findings from the pre-test, including those obtained from discussions with interviewers after the pre-test fieldwork concluded.

  • Questionnaire:
    This section is the main output of the Pre-test Report. The use of the table below is recommended. Please add all modules in the questionnaire. Make sure that all suggested changes are listed and that evidence is provided for final decisions. Please include observations on all country-specific modules and questions.

  • Instructions:
    Describe and list any changes or additions required in the Instructions for Interviewers as well as those introduced in the Instructions for Supervisors and Editors. Such changes typically involve translation issues, instructions for country-specific questions, but also for country-specific response categories. Appropriate corrections are incredibly helpful and will especially inform the main field work training. Note that instruction can be introduced as additionnal hint within each question.

  • Average duration of interviews:
    Calculate the average duration of interview for each questionnaire using the data collected in the pre-test. Typically, as interviewers become more familiar with the tools, this time will decrease and therefore a realistic duration should be proposed and included in the introductory sentences on the cover pages of the questionnaires.

  • Interview process considerations:
    Describe and address the observations from the pre-test that relate to interviewing that will be relevant for training and monitoring in the main field work (for example issues in approaching households, dealing with sensitive module and questions, flow of field work, roles and responsibilities, etc.)

  • Assessment process considerations:
    Describe here the observations, suggestions, and decisions related to the assessment planning and next steps for finalising the questionnaire (training contents/agenda, logistics, staff, support, etc.)