QUESTIONNAIRES UNIT

The QuADEM project has been funded with support from the European Commission. This document reflects the views only of the author, and the Commission cannot be held responsible for any use which may be made of the information contained therein.

!T"#D$CT #!
%or a complete &uestionnaires assessment unit with operatin' instructions, scorin' manual, methodolo'ical compendium, 'lossary and biblio'raphy, please consult this chapter in the QuADEM handboo(.

A))E))ME!T $! T
QUESTIONNAIRES
n 'eneral terms a *&uestionnaire+ is a research instrument consistin' of a list of &uestions that a number of people are as(ed so that information can be collected about somethin'. Most often this method of data collection is used to 'ain statistical data that can serve as the basis for scientific research. ,ut &uestionnaires can also serve a peda'o'ical purpose. $sed in di'ital learnin' modules, &uestionnaires can contribute to an optimal learnin' e-perience in different ways. A &uestionnaire can be used to introduce a certain topic, to arouse the learner+s interest in the subject and to create awareness about the learner+s own views on the issue at hand. Questionnaires can also be used to assess the learner+s .prior/ (nowled'e about a certain topic, allowin' the learner to discover his0her stren'ths and wea(nesses or to assess his0her pro'ress. This, in turn, can motivate the learner or can help him0her to reorient learnin' efforts. n sum, &uestionnaires can increase the interactive &ualities of a di'ital learnin' module. ,y as(in' the learner specific &uestions about a certain topic .and by providin' feedbac(/ they can ma(e the module+s users more motivated, conscious and 'oal1oriented learners. ,ecause of their 1 underrated 1 peda'o'ical potential, a QuADEM assessment unit has been devoted to the use of &uestionnaires in di'ital learnin' modules. The Questionnaires $nit provides you with a tool to assess the methodolo'ical thorou'hness of the &uestionnaires used in a di'ital learnin' module.

)$MMA"2
Definition A questionnaire is a research instrument consistin' of a list of &uestions that a number of people are as(ed so that information can be collected about somethin'.

Scope

This unit can be used for all (inds of &uestionnaires. 3lus0Minus Method Thin( Aloud 3rotocol nterview

Methods

Respondents

"epresentative end user Methodolo'ical e-perts

3"EC#!D T #!)
!o specific preconditions need to be ta(en into account for this unit.

C5EC67 )T
To assess the e-amples in the di'ital learnin' module, you can use the criteria listed in the chec(list below. $se the plus0mius method, a thin( aloud protocol and interviews as research methods to determine to what e-tent, accordin' to the respondents, each statement applies for the di'ital learnin' module at hand. Then continue by chec(in' the bo-es with the scores that reflect the respondents+ opinions best. 4 8 a'ree entirely 9 8 tend to a'ree : 8 neither a'ree nor disa'ree 19 8 tend to disa'ree 14 8 disa'ree entirely !0A 8 not applicable Questionnaires 14 9 4 ; < = > ? @ A 9: The objective of the &uestionnaire is clearly stated. The tar'et audience of the &uestionnaire is well1defined. The &uestionnaire is adapted to the tar'et audience. The &uestionnaire is useful within the learnin' process. The &uestionnaire has a clear and meanin'ful title. The &uestionnaire has a clear layout. The &uestion se&uence is optimal. The &uestionnaire is internally consistent. The &uestionnaire has a reasonable len'th. The &uestionnaire respects respondents+ privacy. 19 )cores : 9 4 !0A

99 94

Questions are not double1barreled. The answer cate'ories of closed &uestions are mutually e-clusive. The answer cate'ories of closed &uestions are e-haustive. Questions are written in clear terms .no jar'on or technical terminolo'y/. Questions don+t contain Bred fla'C terminolo'y. Questions are not presti'e biased. Questions are not leadin'. Questions are consistent with the provided responses. Questions are consistent with the provided responses.

9;

9<

9= 9> 9? 9@ 9A

Additional commentsD

)C" 3T
!e-t you will have to desi'n a script for your assessment sessions. The overview below describes the possible course of an assessment session, summin' up the different possible steps you will have to ta(e as well as offerin' su''estions on how you can smoothly incorporate the different research methods into one session.

!T"#D$CT #!
)tart by briefly informin' the respondents about the set1up of the assessment session and the different research procedures they will participate in.

T5 !6 A7#$D
As( the participants to carry out a limited number of specific tas(s while thin(in' aloud, and readin' aloud the web te-t. n this case you should instruct them to fill out the &uestionnaire.s/ available in the di'ital learnin' module. 2ou can record the audio and0or video of the session as a bac(1up. This method allows you to 'et a picture of the mental mappin' process of the participants. <

37$)0M !$) MET5#D

As( the participants to attentively 'o throu'h the &uestionnaire.s/ available in the di'ital learnin' module. At each &uestion or at each part of the &uestionnaire they value ne'atively, they should note down a minus .1/, at each &uestion0part they value positively they should note down a plus .E/.

!TE"F EG
Ghen the participants have finished mar(in' the te-t with plusses and minuses, you can discuss the mar(ed te-t. 2ou can continue with a short retrospective and structured interview, based on the chec(list. All criteria of the chec(list should be addressed. )core the different criteria to'ether with the respondent. After all respondents have parta(en in your assessment, you can calculate the avera'e scores and use these to score the unit+s chec(list.

MA!$A7
This manual provides you with additional information on the meanin' and the scope of specific criteria and can also offer advice on how to improve certain shortcomin's within the di'ital learnin' module. 2ou can consult it throu'hout the entire assessment procedure. Questionnaires Criteria 9 The objective of the &uestionnaire is clearly stated. E-planation Developin' a &uestionnaire without a clear objective or purpose in mind, will probably result in includin' useless &uestions and omittin' important issues. 5ence, they are a waste of your time and your participants+ efforts. Moreover, a clearly stated objective will increase the participants+ motivation to answer &uestions and allow for &uic(er processin' of results. ; The &uestionnaire is adapted to the tar'et audience. The &uestionnaire is useful within the learnin' process. n order to increase response rates, it is vital to ma(e sure that both the &uestions in the survey and the overall style and lan'ua'e are adapted to the tar'et audience. %or &uestionnaires to be truly successful, it should be inte'rated in the learnin' process. Thin( about timin', method, 'oal, type of &uestions and conse&uences of the &uestionnaire and how the results will be used to enhance learnin'. A &uestionnaire with a title is 'enerally perceived to be more credible than one without and addin' a 'oal is useful for motivation.

<

The &uestionnaire has a clear and meanin'ful title.

>

The &uestionnaire has a clear layout.

Ma(e sure that the presentation of the &uestions and the use of white space, colours, pictures, charts, or other 'raphics do not affect your participants+ interest nor distract from the &uestions. Ma(e sure the answer to a &uestion is not influenced by previous &uestions. 7o'ically 'roup &uestions to'ether. nsert clear headin's. Ma(e sure &uestions flowD lo'ically from one to the ne-tH from the more 'eneral to the more specificH from the least sensitive to the most sensitiveH from factual and behavioural &uestions to attitudinal and opinion &uestionsH from unaided to aided &uestions.

The &uestion se&uence is optimal.

The &uestionnaire is internally consistent.

nternal consistency is the e-tent to which tests or procedures assess the same characteristic, s(ill or &uality. t is a measure of the precision between the observers or of the measurin' instruments used in a study. This type of reliability often helps researchers interpret data and predict the value of scores and the limits of the relationship amon' variables. %or e-ample, a researcher desi'ns a &uestionnaire to find out about colle'e students+ dissatisfaction with a particular te-tboo(. AnalyIin' the internal consistency of the survey items dealin' with dissatisfaction will reveal the e-tent to which items on the &uestionnaire focus on the notion of dissatisfaction.9

The &uestionnaire has a reasonable len'th.

There are no universal a'reements about the optimal len'th of &uestionnaires. t probably depends on the type of participants. As a 'eneral rule, short simple &uestionnaires usually attract hi'her response rates than lon' compleones. 5owever, some studies have shown that the len'th of a &uestionnaire does not necessarily affect response. More important than len'th is &uestion content. 3articipants are more li(ely to respond if they are involved and interested in the &uestion topic. nform participants about confidentiality issues, the status of the survey .voluntary or mandatory/ and any e-istin' data1 sharin' a'reements with other or'aniIations. Double1barreled &uestions combine two or more issues in a sin'le &uestion. They are difficult to answer and ambi'uous

9 :

The &uestionnaire respects respondents+ privacy.

9 9
1

Questions are not double1 barreled.

Writing Guides. Colorade State University. On: http://writing.colostate.edu/guides/research/relval/com2a .c!m >

for interpretation. f double1barreled &uestions occur in the &uestionnaire, rephrase them to separate the issues. E.'. BDo you thin( ta-es should be lowered and schoolin' should be freeJC should becomeD BDo you thin( ta-es should be lowerJ+ and *Do you thin( schoolin' should be freeJC 9 4 The answer cate'ories of closed &uestions are mutually e-clusive. A respondent+s answer should fit in only one cate'ory. E.'. not mutually e-clusive cate'ories areD :19: 9:14: 4:1;: This should becomeD 7ess than 9: 9:19A 4:14A More than 4A 9 ; The answer cate'ories of closed &uestions are e-haustive. All possible responses should be provided. Add more answer cate'ories, or an Bother, please specifyDC, Bdon+t (nowC or refusal cate'ory. A bad e-ampleD BMarital statusD sin'le, marriedC A 'ood e-ampleD Bmarital statusD married, widowed, divorced, separated, livin' to'ether, sin'le, otherD please specifyC 9 < Questions are written in clear terms .no jar'on or technical terminolo'y/. ,ad e-ampleD BDo you believe that the $6 should have a bicameral parliamentJC Kood e-ampleD BDo you believe that the $6 should have upper and lower houses of parliamentJC Gords with emotional connotations or that coincide with stron'ly1held values evo(e emotional responses and may s(ew results. This should be avoided. ,ad e-ampleD BDo you thin( it is fair to murder innocent whales in the 3acificJC Kood e-ampleD B5ow do you feel about whale huntin' in the 3acificJC 9 > Questions are not presti'e biased. "espondents may answer on the basis of their feelin's toward the presti'ious person of 'roup rather than addressin' the issue.

9 =

Questions don+t contain Bred fla'C terminolo'y.

,ad e-ampleD BAccordin' to a recent ACM C5 poll, @: percent of the people oppose copyri'ht protection for the loo( and feel of user interfaces. Ghat is your opinion on this issueJC Kood e-ampleD BGhat is your opinion on copyri'ht protection for the loo( and feel of user interfacesJC 9 ? Questions are not leadin'. 7eadin' &uestions are actually statements dis'uised as &uestions, and ma(e respondents feel that only one response is le'itimate. ,ad e-ampleD BDon+t you a'ree that the loo( and feel of user interfaces should not fall under copyri'ht protectionJC, or BDo you a'ree with the majority of people that the health service is failin'JC Kood e-ampleD BGhat is your opinion on copyri'ht protection for the loo( and feel of user interfacesJC, or BGhat is your opinion on the health serviceJC 9 @ Questions are consistent with the provided responses. ,ad e-ampleD QD 5ow often do youL AD yesterday, last wee(, etc. Kood e-ampleD QD 5ow often do youL AD #nce a day, once a wee(, once a month

)C#"E
After completin' the chec(list, please rate the module on a scale from A to E and e-plain your mar(. %or more information on the scorin' procedure, please consult the scorin' manual. #verall jud'ement )core A , C D

E Assessor+s comments