ITS NOT WHAT YOU ASK, BUT HOW YOU ASK IT

Go Beyond Basic Behavioral And Analytics Data By Using Surveys To Really Understand The Who And The Why Of Your Site Visitors

AN ORANGE PAPER BY

ORANGE PAPER RESPONSE SCALES

TABLE Of CONTENTS
PART I: WHAT IS A SCALE? PART II: WHY USE A RESPONSE SCALE? PART III: CHOOSING A RESPONSE SCALE - GRANULARITY PART IV: CHOOSING A RESPONSE SCALE - SCALE POINTS PART V: CHOOSING A RESPONSE SCALE - BAGGAGE & BALANCE CONCLUSION ABOUT CROWD SCIENCE 4 5 6 8 10 12 13

info@crowdscience.com crowdscience.com 650.625.9722

2011 Crowd Science. All Rights Reserved. 2

ORANGE PAPER RESPONSE SCALES

Online Surveys. Everyone sees them and some even take them. Far less care about what happens with the results. At Crowd Science, we help create great online surveys. We enjoy taking them, too. And we certainly care about the results we work with some of the most well known online companies to make sense of these results so companies are able to take action on the information. The most important aspect of managing and growing an online content site is to consistently ask the right questions to the right people. The right questions provide the power to gain unique insight and then draw accurate conclusions about: Website performance Site satisfaction Visitor demographics Audience growth Monetization opportunities

The most effective and efficient way to influence these elements is to gain a deep demographic level understanding of a particular website audience. This position paper has been developed to shed some light on the importance of creating a high quality survey questionnaire. Since the inception of the Web, the Crowd Science team has had extensive online survey experience and learned a lot about survey best practices. One of the most important aspects of a survey questionnaire is the response scale. The response scale has a key role in the type and quality of the analysis that can be drawn from your survey research. Sandra Marshall, Crowd Science VP of Research, explains why the response scale matters.

info@crowdscience.com crowdscience.com 650.625.9722

2011 Crowd Science. All Rights Reserved. 3

ORANGE PAPER RESPONSE SCALES QUESTIONNAIRE DESIGN: RESPONSE SCALES

Questionnaires, in market research anyway, are somewhat like the eyes a window to the soul. That may sound a bit lofty, but we rely heavily on questionnaires to access information on peoples attitudes and behavior. A well-designed questionnaire is vital to getting to the truth. A poorly designed questionnaire will get you answers just not necessarily the right ones. Its that old adage of GIGO (garbage in, garbage out) after the initial stages of research and sampling design have been addressed, its the questionnaire that can make or break the research. There are many aspects of questionnaire design, so in the interest of starting somewhere, lets begin with a topic that incites a lot of discourse and debate: Scales specifically, response scales. Discussions about other types of scales will best be addressed in future position papers.

PART I: WHAT IS A SCALE?

Similar to scales used to measure things (like weight or height), surveys employ scales to provide a range or degree of information. Using a scale in a survey provides those who take the survey with a range of responses for each question providing a richer understanding and level of detail.

RESPONSE SCALE
Very unlikely 1

Somewhat unlikely 2

Neither likely nor unlikely 3

Somewhat likely 4

Very likely 5

info@crowdscience.com crowdscience.com 650.625.9722

2011 Crowd Science. All Rights Reserved. 4

ORANGE PAPER RESPONSE SCALES PART II: WHY USE A RESPONSE SCALE?
A key aspect of quantitative, closed-ended questionnaire design involves not only how the question is worded, but also how we design the response categories. Why ask one question to get one response, when you can ask one question to address several items? For example: Instead of asking whether a respondent plans to buy a new car in the next 12 months, ask the degree of likelihood to buy within the next 12 months. Rather than asking whether a respondent is merely satisfied or dissatisfied with a product or service, ask their level of satisfaction or dissatisfaction.

Scales that go beyond this or that response options open up a new set of possibilities to better understand respondents. The proper use of response scales can inform where risks lie and provide the opportunity to be in a better position to minimize them. Shades of Grey A response scale offers more detail than a simple black or white type of response. It digs into the range of space between the black and the white: Yes or No? Do You or Dont You? Will You or Wont You?

A scale is almost like having a dimmer switch the respondent can dial up or dial down the intensity of their response. You end up with more detail on the degree of activity, intent, or attitude. You also have more options when it comes time to conduct the statistical analyses. With this level of detail, smarter decisions can now be made on how or where to allocate resources and efforts to maintain and grow customer bases/audiences/etc. So next time you catch yourself asking whether, see if the question translates into a scale format that now allows you to ask how much. Itll go a long way in providing a full picture, and better answers.

info@crowdscience.com crowdscience.com 650.625.9722

2011 Crowd Science. All Rights Reserved. 5

ORANGE PAPER RESPONSE SCALES PART III: CHOOSING A RESPONSE SCALE - GRANULARITY
Once youve decided to use a scale, you are now faced with the decision of what kind of scale to use, and what it should look like. There are many ways to present response scales, and each one has its particular strengths and purpose. More Points = Greater Sensitivity As with everything in research, always go back to the objectives: What is the desired goal (or research hypothesis)? How do you intend to use the results?

This will drive every aspect of questionnaire design, including the appropriate response scale to use. For example, lets say you want to understand customer satisfaction to track changes between specific time periods which are fairly short (hourly, perhaps). You may therefore want the response scale to be more sensitive in order to pick up on the smallest of changes. This will require the use of a more finely tuned scale - one with more points on it. This would enable you to act nimbly on any change that was recorded from hour to hour. But perhaps you may not want to be bothered by minute fluctuations, in cases where they may be distracting, less meaningful, or in some way not worthwhile to act upon. Perhaps the measurement normally fluctuates a great deal, and you need to obtain a sense of the bigger picture, where fluctuations go beyond the norm. In these cases, a smaller number of scale points might be in order. Again, there may be other factors affecting your decision when choosing the right scale range. Even or Odd? When deciding on how many points to include on the scale, another decision to make is whether to have an odd or even number of points. An odd number provides a midpoint, which may or may not be desirable, depending on your objectives, and how you intend to use the information. You may not want to use an odd number in order to avoid a neutral response. For example, perhaps you are measuring satisfaction, and you really want to know whether your customers are satisfied you want to force the issue and get them to choose a positive or negative rating, without sitting on the fence. You may decide on a scale that includes the following even number of points:

info@crowdscience.com crowdscience.com 650.625.9722

2011 Crowd Science. All Rights Reserved. 6

ORANGE PAPER RESPONSE SCALES

EVEN NUMBER OF POINTS

Very dissatisfied 1

Somewhat dissatisfied 2

Somewhat satisfied 3

Very satisfied 4

By offering no point between dissatisfied and satisfied, you force the respondent to commit to one side or the other, presumably in order to know whether/how much you need to improve your offering. However, lets say youre asking about some hypothetical future behavior, such as intent to purchase. Its possible that the respondent truly does not know whether they will carry out this behavior, because they require more information (costs, benefits, their future needs/situation, etc.) before making a decision. Similarly, asking opinions on topics the respondent has never before encountered, means it is possible that they have not yet formed any opinion on the subject at all. In such cases a midpoint will offer a position where they can respond without committing to a positive or negative answer.

ODD NUMBER OF POINTS

Very dissatisfied 1

Somewhat dissatisfied 2

Neither satisfied nor dissatisfied 3

Somewhat satisfied 4

Very satisfied 5

Alternatively, if an even-numbered scale is offered in addition to a separate Dont know category, this could achieve the same result. Either way, its important to be sensitive to situations where forcing an answer one way or the other may be inappropriate. In the end, when deciding on the number of points in your scale, think ahead to the analysis and how you want to present results, think about the nature of the measurement youre taking, and think of the respondent experience in answering your question.

info@crowdscience.com crowdscience.com 650.625.9722

2011 Crowd Science. All Rights Reserved. 7

ORANGE PAPER RESPONSE SCALES PART IV: CHOOSING A RESPONSE SCALE - SCALE POINTS
Anchors Aweigh! So far, weve addressed why to use a response scale, and some things to consider in deciding the number of points in the scale. Now lets discuss how to label the scale points. A Picture Is Worth A Thousand Words Some scales dont even use verbal descriptions, and rely on diagrams to conveying meaning, such as simple line drawings of faces with various expressions (e.g., big smile, medium smile, straight-line mouth, medium frown, and big frown). These types of scales may be successful with audiences (like children) that are preliterate, or for whatever reason have difficulty expressing themselves with words.

SCALE USING DIAGRAMS

Such situations may require an interviewer to assist in administering the questions if this is the case. However, even with respondents able to read the questionnaire, such scales may make the experience more fun or engaging, depending on the situation. Too Many Words Spoil The Broth Most scales use words to define their points. You might consider putting verbal definitions on every point in the scale (a.k.a. fully anchored), or to use a partially anchored scale, say, only defining the two opposite ends. Some people wonder about labeling the two ends, plus a label for the midpoint if there is one. A general rule of thumb is to fully anchor or label a scale if there are five or fewer points on it, and for scales with six or more points, a bipolar anchor can do nicely.

FULLY ANCHORED SCALE

Somewhat Very dissatisfied dissatisfied 3 1

Neither satisfied nor dissatisfied 4

Somewhat satisfied 5

Very satisfied 7

PARTIALLY ANCHORED SCALE

Very dissatisfied 2 1

Very satisfied 7 6

info@crowdscience.com crowdscience.com 650.625.9722

2011 Crowd Science. All Rights Reserved. 8

ORANGE PAPER RESPONSE SCALES

There may be cases where scales using six or more points can still be completely anchored on all points, but in some cases this can become tedious for respondents to read through (or in telephone surveys, to have it read out loud for every question). On self-completed surveys, having so many labels may compromise the visual appeal, and the questionnaire may appear too cluttered, long, or confusing for the respondent.

FULLY ANCHORED 7 POINT SCALE

Somewhat Very Mostly dissatisfied dissatisfied dissatisfied 3 1 2

Neither satisfied nor dissatisfied 4

Somewhat satisfied 5

Mostly satisfied 6

Very satisfied 7

Twos Company, Threes A Crowd You may wonder if it is helpful, when using bipolar anchors on each end of a scale, to also label the midpoint to provide another point of reference for the respondent. You are arguably introducing risk by calling unnecessary attention to the midpoint. This emphasis on the midpoint may tempt respondents to use it as an out from having to think about their response and provide an opinion. Midpoints are also tempting fallback for lazy or fatigued respondents, so the less attention to the halfway point in the scale, the better the results.

info@crowdscience.com crowdscience.com 650.625.9722

2011 Crowd Science. All Rights Reserved. 9

ORANGE PAPER RESPONSE SCALES PART V: CHOOSING A RESPONSE SCALE - BAGGAGE & BALANCE
Lose The Baggage, Keep The Balance When constructing a response scale, its important to achieve a nice variety of response options across the scale. A scale that tends to consistently elicit responses skewed toward one very small part of the scale doesnt tell you much. It would be like measuring agreement with the statement adequate sleep contributes to good health. Most people would agree this statement is true; there would probably be minimal variability in the results, either across time or across respondents. On A Scale Of One To Ten In many cases, the nature of the question causes responses to skew a certain way. But sometimes it is the nature of the scale that skews the data in a particular direction. For example, consider 10-point scales, e.g. On a scale of one (or zero) to ten, please rate your agreement/satisfaction/etc. with the following statement. The potential problem with a 10-point scale is that it carries a lot of baggage from prior experience. This baggage can include familiarity with other questionnaires that have used this scale. It also can stem from the popularization of various commonly used 10-point rating or ranking systems. For example: David Lettermans Top 10 The movie 10 [Hello, Bo!] School tests scored out of 10 Casual discussions On a scale of one to ten, what did you think of that movie?

You may be saying, So what? These examples demonstrate how respondents are already conditioned look at 10-point scales in a predetermined way. For instance, think of those tests or quizzes that everyone took as kids in school, graded out of ten. Here we have been conditioned to focus on the upper end of the scale. Specifically, most of these tests required at least a 5 to pass. Anything below 5 was a fail. It didnt really matter by how much you failed, either - after all, if you failed the test, you failed! There is little importance or meaning attributed to the differentiation of any scale points below 5. But for points 5 and upward, we attribute differences for each of the scores, even corresponding the numbers to different letter grades, A through F. This is one situation that can be conditioned into respondents from a fairly early age and contribute to our reluctance to use the lower part of the 10-point scale.

info@crowdscience.com crowdscience.com 650.625.9722

2011 Crowd Science. All Rights Reserved. 10

ORANGE PAPER RESPONSE SCALES

If You Dont Have Anything Nice To Say. There is another piece of baggage respondents can carry around: the desire to be nice. Dont get me wrong - were not knocking politeness as a principle. But in the case of questionnaires, the desire to comply with socially acceptable practices can be detrimental to survey results. Negative feedback is very necessary, because without it we dont have the means to improve! Nevertheless, many people are reluctant to say bad things because its not the nice thing to do. And if anything lower than the midpoint of a scale constitutes a failing grade then that would be a pretty negative piece of feedback. Lucky Seven At Crowd Science, when the situation is appropriate, we prefer to use a 7-point scale, which offers granularity without baggage. Instead of the overused 10-point scale, we throw the respondent off the track and get them to think about things with a fresh approach and without any baggage. Seven point scales offer a midpoint (for cases where we dont want to force a positive or negative rating) and enough granularity to track movement from one situation or time to another. And best of all, there is less bunching of responses that you would expect to see in the 10-point scale.

7 POINT SCALE
Very dissatisfied 2 1

Very satisfied 7 6

info@crowdscience.com crowdscience.com 650.625.9722

2011 Crowd Science. All Rights Reserved. 11

ORANGE PAPER RESPONSE SCALES CONCLUSION

Its one thing to ask a question. Its equally, if not more, important to have the right scale to get responses that provide actionable data. Responses can be far more meaningful by making a few simple tweaks to the response scale. The newly exposed grey areas in the response set have tremendous value. By knowing how much or little grey area your questionnaire responses have will alter the conclusion you can draw for the question. The number of options to choose from within a scale can also be a huge factor in the type of response youre looking for. An odd number of responses or a dont know option will allow the respondents to take a neutral stand, while an even number of responses will force them to pick a side. The number of options in the response set, along with the way the response options are worded, will drive the type of meaningful responses that youre looking for. Remember, its not what you ask, but how you ask it.

info@crowdscience.com crowdscience.com 650.625.9722

2011 Crowd Science. All Rights Reserved. 12

ABOUT CROWD SCIENCE

Crowd Science Inc. (http://crowdscience.com) is revolutionizing online market research by providing innovative solutions and technologies to online publishers and brands. Companies like Federated Media, Turner Network and Everyday Health use Crowd Sciences combination of web analytics and survey research to improve marketing and advertising precision. Contact us today at info@crowdscience.com or 1-650-625-9722.

OUR PRODUCTS
Get accurate intelligence about audiences that cant be uncovered using standard analytics, behavioral inferences, or low quality surveys. Built on SmartSample technology Questionnaires pre-built by our researchers Rich & statistically valid results

AUDIENCE PROfILER

AUDIENCE TARGETER

Audience Targeter provides publishers and networks with direct paths to segmenting, targeting and monetizing their audiences. Measurement collect deep, customized audience data Segmentation organize audiences into sellable subgroups Optimization bridge segmentation metrics with ad serving Extension grow frequency & reach via re-targeting

CAmPAIGN mAxImIzER

Delivering real-time results that allow you to understand and report key brand campaign metrics: Brand Lift favorability & usage Awareness aided & unaided recall Intent purchase intent

SITE SATISfACTION

Site Satisfaction research provides publishers with actionable information about their audiences. Services include: Full service program management Cost-effective research & actionable deliverables Innovative technology & real-time dashboards

ONLINE mARKET RESEARCH

Full suite of traditional custom online research services, including: Research consultation & design Questionnaire design & programming Data collection & fielding Reporting, analysis, & insights
2011 Crowd Science. All Rights Reserved. 13