Being able to compare alternatives is essential when designing an evaluation. This post looks at some alternatives to transcribing interviews.
Last week a colleague asked me for some advice about her planned evaluation. She has just completed a pilot project which has been very successful. Now she wants to run the project again. The pilot project had extensive evaluation activity, including interviews with all participants. These interviews were very useful both on terms of the information it provided and the value of the reflective process itself for the participants. For the ongoing program, she wants to continue to conduct the interviews, because of the value of the process, but needs an alternative to recording and transcribing the interviews because the cost would be too much. What alternatives are there?
This is a common challenge in evaluation. While interviews are often used, often it is not possible to record and transcribe them for analysis: sometimes because of the cost of transcription or restrictions on recording the interviews sometimes waiting for a transcription will not provide timely information.
A common strategy is for the interviewer to write notes during the interview, and then use these to write up the interviews. But there are considerable risks to validity in this strategy, especially for rich, semi-structured interviews which are not simply going through a list of questions and writing down short answers. There is no scope for transparency and review of what has been recorded, so its accuracy depends on how well the interviewer can simultaneously engage in an interview and take notes, the accuracy of their understanding of what is being said (harder in cross cultural interviewing) and their ability to overcome cognitive biases, such as finding it easier to remember things that fit with one's own views.
So are what are some reasonable alternatives to transcription? Here are three I suggested to my colleague:
Sharing a summary of the interview with the interviewee at the end of the interview
At the end of the interview, take a few moments to go back through your notes with the person or people who have been interviewed, and say something like, "Do I have this right? These seem to be the main issues you've raised…" and list them. This can be especially useful for interviews which have covered a lot of issues. For example, in a recent project, I was interviewing program managers about their experience of commissioning and managing an external evaluation. While I had specific questions to ask, I was mostly trying to understand their perspective on the experience, so they were encouraged to talk freely about their experiences and raise issues that were not on the interview schedule. In one interviews, when technical difficulties made it impossible to record the interview as planned, I made sure at the end of the interview to summarise what I understood to be the main points they were making, and invited them to correct or elaborate on it.
Doing a graphic recording of the discussion
This can be particularly helpful in a group interview but can also be used in individual interviews. A graphic recording develops a visual summary of the conversation, such as showing different components, or alternatives, or dichotomies, or sequences. This can help others to share an emerging picture of what is being discussed, and then participants can be asked to validate, challenge or extend it. For a group interview, this can be done on a flip chart or a large sheet of paper along a wall. I sometimes do this on a sheet of paper when doing an individual interview.
Using a smart pen to provide audio of critically important sections
A smart pen records audio synchronised to notes taken during the interview. This can significantly reduce the time needed to write up a good account of an interview. Most of the interview can be written up from the notes. Where there is a need to check details or get exact quotes, simply touching the pen to that section of the notes plays the audio. This makes it easy to transcribe only key sections of the interview. If there is a need to check the whole interview, the audio file can be downloaded, shared and transcribed. A transcription service can provide a transcript of the file if needed. I have used a LiveScribe smart pen for interviews and it's been very useful, but it only works on special paper (which can be bought or printed out), and the smart pen needs special cartridges which are not always readily available, and of course it needs to be charged up and connected to computers. While it worked well while I was using it, the version I was using was not so easy to pick up and use again after a break.
What other alternatives to transcribing interviews have you used? Do you have other suggestions, or experiences of using the alternatives described above?
Image: Scribe, by jefka on Flickr