Telephone Questionnaires

Surveying is a way to collect information directly from project stakeholders, participants or beneficiaries in a systematic, standardised way, and rely on the use of questionnaires distributed to respondents. These questionnaires can be delivered through a variety of options, including postal surveys, face to face interviews, using handouts, online forms, or by telephone. 

Telephone surveys are most appropriate when:

  • Results are needed quickly
  • The target sample have telephones (not appropriate, for example, in rural parts of developing countries, unless mobile phone usage is widespread)
  • Members of the sample would have difficulty completing a written questionnaire
  • Survey questions are clear and easily comprehensible
  • The survey is relatively short
  • Investigators have the resources (staff and money) to make the necessary calls

The advantages of telephone surveys include the fact that callers can provide additional information and clarification, or can ask probing, clarifying questions. Telephone surveys also tend to have higher response rates than paper equivalents, meaning self-selection bias can be better controlled. On the other hand, as mentioned, they require respondents to possess phones (and carry the potential of bias if investigators are excluding non-phone owning individuals in a sample), and demand language and communications skills from those administering the survey.

At the evaluation stage surveys – of any form - are particularly useful tools for either triangulating other evidence about the impact and effectiveness of a project or programme, providing information on how the project’s effectiveness is perceived by stakeholders, or to gather evidence about the real impact on people’s lives.

When planning a telephone survey, investigators should:

  • Determine whether a telephone survey is the most appropriate data collection option.
  • Create a project plan, including a budget, timelines and responsibilities
  • Decide who should be involved in the survey design, and consult them fully.
  • Define the survey content. What do you need to know and how will you use the data?
  • Indentify respondents: How large should the sample be and how can you ensure representativeness? Sample size depends on what you hope to do with the data. To generate frequencies and percentages, smaller samples can be used than if you wish to use regression analysis.
  • Develop the questionnaire: Telephone surveys should be short with easily understandable questions. You should cover the most important information first. Develop a standard ‘script’ for callers, explaining who they are and the purpose of the survey. Establish guidelines for callers, including anticipated questions or issues they may encounter. Train the callers.
  • Prepare materials and facilities for making the calls, including: telephones, scripts, online forms, call logs etc. Consider providing advance notice to participants or scheduling interview times with them in advance.
  • Pilot the questionnaire to ensure the wording is clear, to check the time the survey takes and that it yields the right kind of data. Refine accordingly.
  • Conduct the full survey. Capture results rigorously using call logs and databases.
  • Analyse data and communicate findings

Source: Taylor-Powell, E. and Hermann, C. (2000) “Collecting Evaluation Data: Surveys”, University of Wisconsin.

Brief Example

The UK’s public sports-promotion organisation, Sport England, launched its ‘Peer-led Improvement Pilot Project’ in 2007. The project was designed to encourage local authorities to continuously improve their sport and recreation services. It commissioned an evaluation of the pilot project. The evaluation used a variety of methodologies including gathering reaction data straight after training courses, a telephone interview with participants and a final evaluation workshop.

The evaluators designed the telephone questionnaire and set up times for calls in advance. This allowed participants to think about their answers. Calls lasted about 45 minutes and structured questions around the project goals. Interviewees were allowed to email any further thoughts to evaluators after the telephone call. The evaluators used the surveys to draw out key quotes in their final report, and to draw conclusions and recommendations for the roll-out of the project.

Source: Sport England (2007) “Working together for Excellence: Peer led Improvement Pilot, Evaluation Report, Appendix 4”. Online.

Advice

Advice for CHOOSING this option (tips and traps)

  • A survey is of greatest use when evaluation questions are best answered by stakeholders themselves, particularly so when attempting to measure changes in behaviours, attitudes or beliefs.
  • If anonymity is likely to be important to respondents, then an anonymised paper survey may yield more honest results than a telephone survey. Likewise, some more complicated questions are better understood on paper.
  • Telephone surveys are most useful of you need results quickly, or in developing country contexts where many people may be illiterate. However, telephone surveys suffer from other selection biases (including phone ownership) and are more expensive than other alternatives.

Advice for USING this option (tips and traps)

Interviewers conducting telephone surveys should always:

  • Adopt a professional tone, and be able to clearly articulate their questions.
  • Deliver each question exactly as is written and ordered.
  • Be guarded against over-empathising with their respondents or offering any unsolicited comments that might bias responses.
  • B respectful of the respondent’s wishes – including their requests to end the survey part-way through or reschedule for a more convenient time.
  • For open-ended questions (i.e. “how do you feel about X”), it is important to record, verbatim, the exact wording of the respondent’s reply. For particularly long responses that require short-hand note-taking, the interviewer should reserve time at the end of survey to clean up their notes.

Resources

Guide and tool

Taylor-Powell, E. and Hermann, C. (2000) “Collecting Evaluation Data: Surveys”, University of Wisconsin. - Concise guide for general and telephone survey design.

Lavrakas, P. (1993) Telephone Survey Options: Sampling, Selection, and Supervision, 2nd ed. (London: SAGE, 1993)

Example

Sport England (2007) “Working together for Excellence: Peer led Improvement Pilot, Evaluation Report, Appendix 4”. Online.

Sources

Taylor-Powell, E. and Hermann, C. (2000) “Collecting Evaluation Data: Surveys”, University of Wisconsin.

Lavrakas, P. (1993) Telephone Survey Options: Sampling, Selection, and Supervision, 2nd ed. (London: SAGE, 1993)

BIS (2009) “RDA evaluation: Practical Guidance on implementing an Impact Evaluation Framework, Appendix 1: Beneficiary Survey methodology and questionnaires”. Online.

Author

Josiah Kaplan

Updated: 13th January 2014 - 2:51pm
This Option is useful for:

Comments

There are currently no comments. Be the first to comment on this page!

Add new comment

Login Login and comment as BetterEvaluation member or simply fill out the fields below.