Effects of question order on the assessment of police performance

Background
Answering questions in a survey is a complex cognitive process. The question order plays a key role in this process: Preceding questions may activate information that may not have come to the respondents’ minds if other question had been asked. The activated information in turn can influence how respondents answer subsequent questions (“priming”). An example: Asking questions about victimization in the past 5 years may make non-victims realize that they were not victimized in the past 5 years. Realizing this may “prime” the subsequent answers of non-victims resulting for example in a very positive assessment of police performance.
Experimental approach
In the International Crime Victims Survey (ICVS) 2005, the respondents were first asked whether they had fallen victim to different crimes in the past 5 years. Crime victims, who had reported the crime to the police, were asked to rate their satisfaction with the way the police handled the crime. After these specific questions addressing the victims only, all respondents were asked the following question to assess police performance in general “Taking everything into account, how good do you think the police in your area are at controlling crime?“ We wondered whether this question order affected the assessment of police performance.
To discover a potential question-order effect, a split-ballot experiment was conducted within a follow-up of the ICVS – the Luxembourgish “Enquête sur la sécurité 2013“ (N = 3025). Half of the respondents were asked the general question regarding police performance at the beginning of the questionnaire before the questions concerning victimization and victimization details (Group 1). The other half answered the general question after the specific questions (Group 2).
Results
Respondents in group 2 (general question at the end) were less likely to choose the extreme categories “very good job” (G2: 6.7% vs G1: 11.5%) and “very bad job” (G2: 1.3% vs G1: 2.6%) and instead were more likely to choose “don’t know” (G2: 7.7% vs G1: 3.4%; Sig. <.001).
Furthermore, the question order had a strong effect on the sub-group of respondents who were dissatisfied with the way the police handled a crime. Respondents who had already expressed dissatisfaction (group 2) were much more likely to say the police in Luxembourg are doing a “very good/good job” than respondents with the opposite question order (G2: 62.6% vs G1: 39.8%; Sig. = .002). This result suggests a “contrast effect” of asking the more specific question first; i.e., respondents did not consider the negative information regarding their dissatisfaction with the way the police handled a specific crime when they answered the general question regarding police performance.
Conclusion
The question order affects the assessment of police performance by activating information that is relevant for the assessment. Researcher should conduct split-ballot experiments if they think that preceding questions “prime” subsequent answers.