Research Design and Methods Questionnaire Design

Probability sampling

•       Every case should have an equal chance of being selected

•       Random selection is to create a sample that is similar to the overall population, in order to be able to make inferences about your findings

•       In each of these approaches, all cases are given a number starting at 1

Non-probability sampling

•       Sampling approaches that are not regarded as representative

•       Nevertheless, convenience and quota sampling are frequently used in quantitative research

What else can bias your sample?

Face-to-face surveys: favour ‘friendly looking’, approachable people

Online surveys: favour those who are on email and have ICT skills

Postal and telephone: favour those on ‘official’ lists, e.g. the phone book, businesses registered for tax

Weekdays/Day-time: may exclude those who work

Evenings/weekends: not good for workers or businesses!

Age groups: older people are more likely to take the time to complete a survey

Interest: those with knowledge or interest in the topic may be more likely to respond

Wording Questions in Surveys

1. Use simple, clear questions
2. Avoid using jargon and technical terms
3. Ensure that, if the respondent is not familiar with the topic, you explain it to them
4. Avoid using leading questions

–      Ask questions in a neutral way (not “do you agree that…”)

–      Provide options for people who don’t know or have no opinion

–      Providing a balanced list of answer options

What is wrong with the following questions?

•       What are your views on the recent amendments to the Disability Discrimination Act?

•       Do you agree that petrol prices are too high?

•       Do you agree with fox hunting even though it is cruel and barbaric?

•       Most people in the UK think that the enlargement of the European Union is a bad thing.  What do you think about it?

•       How are you feeling today?

Open and Closed Questions

Closed Questions

A number of possible answers are given for the respondent to choose from

Easy to complete (tick-boxes, scales)

Easy to analyse (answers can be coded and counted – quantitative data)

Force people into false opinions?

Don’t help you to understand reasons for answers

Open Questions

Respondents formulate their own answers

Take more time for respondent to complete

More difficult to analyse (may need to identify themes, and/or use quotes)

Enable respondents to give own opinions in their own language

Produce qualitative data

Question Formats

•       Numerical value data – e.g. How old are you?

•       Binary choice data (tick box) – e.g. Do you drink alcohol? Yes/No 

•       Multiple choice formats – e.g. are you:

•       Student

•       Employed

•       Unemployed

•       Other

•       Rating scales (for attitudes)

•       E.g. likert scales, horizontal rating scales

•       Ranking questions

•       e.g. please select from the following list the three issues you consider MOST important

•       Open-ended questions

•       E.g. Please explain your answer to q. 2 above

Likert scales

•       Good for probing several measures – to gather a lot from one question

•       Easy and quick to complete

•       Often used for measuring attitudes, beliefs, levels of satisfaction

•       A balanced scale that can be better for exploring respondents’ views than simple ‘yes/no’ answer options

•       Can group responses (e.g. combine very and fairly satisfied or calculate the mean, say out of 1-5)

•       Good for comparative analysis, e.g. between user groups or over time

Likert Scales – Examples

To what extent do you agree or disagree with the following statements (please tick

	Strongly Agree	Agree	Neither Agree nor Disagree	Disagree	Strongly Disagree
The food at the café is healthy
The cafe is good value for money
The staff at the café are friendly

•       Please circle as appropriate, where 1=least important and 5=most important

                    1                          2                             3                     4                      5

•       Please mark a cross on the line where best fits

v.good   |             |                |                  |             |        v. bad

Questions to identify priorities

•       Please circle three from the following list that you consider most important

•       Please rank the following 5 options, by placing the numbers 1 to 5 next to each (where 1 is most important and 5 is least important)

•       Please allocate 100 points amongst the five areas below; the areas that you feel to be more important should be given more points. Please ensure that once the points have been delegated they add up to a total of 100.

Sometimes researchers combine measures of importance with measures of satisfaction. This helps ensure that they are paying most attention to levels of satisfaction in aspects that respondents value the most.

Questionnaire Design and Layout

•       Start with easy questions – move to more difficult questions

•       Put more sensitive questions at the end (e.g. age, marital status, income)

•       Put open-ended questions towards the end

•       Use a mixture of question formats – keep it interesting

•       Use filter questions to ensure questions are relevant

•       Don’t make the questionnaire too short or too long!

Validity in survey research

Does our survey help us to answer our original research question or does it provide us answers to a different question entirely?