Questionnaire Design Problem – Mixed Mode Scales
Data from mixed mode questions are hard, if not impossible, to interpret accurately. Here is an example of a mixed mode question and my problem analysis related to interpreting the results this mixed mode scale would produce. Notice how the double concept appears workable, but quickly gets the researcher into trouble as he or she tries to understand the result. In a follow-on article, I will provide a solution to the dilemma posed in the problem analysis provided here.
The Mixed Mode Question Q10. Please rate the effectiveness in achieving your business goals compared to your expectations for each of the following social media networks your company uses. Please use the scale below, where: 1 = much lower than expected and 7 = much higher than expected.
Effectiveness in achieving business goals was:
Much lower than expected 1 2 3 4 5 6 7 Much higher than expected
[Note the items to be rated are not necessary to the example, feel free to fill in your own]
You may already see the problem, but allow me to point it out. The problem is a respondent could score ‘Effectiveness’ for an item ‘higher than expected’ but his or her original expectation was low or score ‘Effectiveness’ for an item ‘lower than expected’ with a starting expectation that was high. Without knowledge of respondents’ original expectation levels the data are nearly impossible to interpret.
To provide clarity we can explore the problem further by looking at four extreme scenarios:
1. If all respondents started with low expectations and their responses were effectiveness in achieving business goals was (1) much lower than expected. What would be your interpretation?
2. On the other hand, if all the respondents had high expectations and they responded effectiveness was (1) much lower than expected. What would be your interpretation?
3. Conversely, if all the respondents started with low expectations and they responded effectiveness was (7) much higher than expected. What would be your interpretation?
4. Once again, if all the respondents start out with high expectations and they respond effectiveness was (7) much higher than expected. What would be your interpretation?
Scenario Analysis Now let us examine each scenario and compare the interpretations that logically follow based on the extreme cases previously outlined.
In scenario # 1, people start out with low expectations and their experience is lower still. Wow! What does this say? Social Media was not expected to do much and it did even less!
In scenario # 2, people start out with high expectations and their experience is lower than expected. Well perhaps their original expectations were unrealistically high! This tells us something entirely different then the first scenario.
In scenario # 3, people start out with low expectations and they experience higher value than expected. Okay, good, but it probably was not hard to meet or exceed the low bar they set – right?
In scenario # 4, people start out with high expectations and they experience value higher than their already high expectations. Wow, now that’s a winner! And, a very different interpretation from any of the other scenarios.
As you can see, mixed mode questions (or scales) are important to avoid. In a longer article published on our website (See, Questionnaire Design Problem – Mixed Mode Scales), we provide a solution to this research problem showing how to fix the serious design flaw illustrated here. The solution explores the relationship between effectiveness and expectations allowing you to interpret the data accurately.
Note: Mixed mode scales are one of the 30 quality control (QC) items checked as part of our standard Questionnaire Audit service.
For more details on sampling read the eBook “Sampling Dilemmas and Solutions.” Find it on http://www.Atheath.com/MRRC Feel free to contact Carey he is a research professional with 20 years experience and two advanced degrees and author of: Questionnaire Design for Business Research (2010, Tate Publishing) http://questionnairedesign.tatepublishing.net/ Read our blog “The Research Playbook” when you visit our website.