EDF5442

Final Paper on Course of

Inquiry & Measurement

The Relationship between the Effectiveness of a Facultys Questioning and the Students Evaluation of the Instructional System Program

Professor: Dr. Shute Student: Meizhen Chen

Date: April 17, 2012

Page 1 / 18

The Relationship between the Effectiveness of a Facultys Questioning and the Students Evaluation of the Instructional System Program

Contents
Cover .......................................................................................................................................................... 1 Introduction ............................................................................................................................................ 3 Literature Review ................................................................................................... 4
Classifying Classroom Questions ............................................................................................................. 4 Using the Research for Effective Supervision: Measuring a Teacher's Questioning Techniques ......... 5 Pre-service Teachers Thoughts about Teachers Questions in Effective Teaching Process ................. 6

Purpose of Using Literature for research

................................................................................. 6

Methods ................................................................................................................. 7
Research Design ....................................................................................................................................... 7 Participants ............................................................................................................................................... 7 Research Instruments .............................................................................................................................. 8 Data Analysis............................................................................................................................................ 8

The Findings and Discussion ........................................................................................................... 9

Perception of Questioning Effectiveness Data Analysis ....................................................................... 11 Evaluation of the IS facultys Questioning Effectiveness ..................................................................... 14

Conclusions ........................................................................................................................................... 17 References

............................................................................................................................................ 18

Page 2 / 18

Introduction
To be or not to be: that is not an effective question! --- Meizhen Chen Questioning plays an important role in daily instructional dialog, during which teachers often endeavor to evoke students thinking by questioning. In turn, students not only respond with answers, but with further questions generated to develop their thoughts. Mainly, such an informative activity is initiated by teachers effective questioning, which serves as a model for students formulating their subsequent questions. As a result, the quality of questions that students ask can reflect the effectiveness of teachers questioning, as well as the quality or level of the students thinking. Even though teachers ask about 120 questions per hour on average (Carlson 1991; Carlson 1997; Graesser & Person 1994), many questions have a low level of effectiveness, which may be due to a lack of knowledge about questioning taxonomies and sequencing, which is essential knowledge for productive verbal questioning (Vogler, 2005). Since so much time is devoted to questioning, its critical for teachers to address the problems of effective questioning and the methods for improvement. So far, the current literature mostly focuses on enhancing teachers knowledge and skills for effective questioning. However, literature rarely discusses the conflicting perceptions about effective questioning between teachers and students. Therefore, teachers should recognize the characteristics of the questions that students value, which can help teachers generate effective questions, and thus cultivate students intellectual growth. Hence, based on this assumption, this research is intended to identify whether there is a conflict between faculty and students perception of effective questioning.

Page 3 / 18

Literature Review
Classifying Classroom Questions (Riegle, 1976) Often times, prior to generating questions, teachers are inclined to determine the type of questions for their target audience. The questions can be easy, tough, concrete, abstract, short or long, which are based on the audience or context accordingly. The subject of this study assumes that classifying classroom questions can be regarded as one of techniques for improving teachers questioning skills. The study compares three cross-subject Question Classification Systems in terms of their advantages and disadvantages via the literature, which are Blooms Question Classification System, Wilsons Statement Classification System, and the Alternative Question Classification System. (Rodney P. Riegle. 1976) First, based on cognitive process, Blooms Taxonomy classifies six types of thinking in a sequence from the lowest level to the highest level, including Knowledge, Comprehension, Application, Analysis, Synthesis and Evaluation. (Vogler, 2005) However, many research studies indicate that most questions generated by teachers in a typical classroom context only reach low levels: knowledge and comprehension. Therefore, this study suggests teachers to examine their questions by using Blooms Taxonomy, so they may recognize the corresponding level of thinking triggered by their questions. Teachers may also utilize Blooms Taxonomy as standards to formulate or modify the type of questions to cultivate students different thinking abilities. However, Rodney (1976) points out that this cognitive-process approach to question classification has its weaknesses: one is that these processes cannot be observed directly, even the same type of questions might arouse different types or levels of cognitive process among students. The other one concerns Blooms categories, which only cover a few important educational objectives, yet there are several other worthwhile question types, such as questions which guide students learning a behavioral or social skill.
Page 4 / 18

The other two classification systems define classroom questions in different aspects, which compliment Blooms Taxonomys weakness. Wilson distinguishes five kinds of statements as: Imperatives and Attitude Statements, Empirical Statements, Analytic Statements, Value Statements and Metaphysical Statements. (Rodney P. Riegle. 1976) Among them, Metaphysical questions are regarded as a useful addition when it involves religions, while the Attitude Statements require answers with justices. On the other hand, the Alternative Classroom Question Classification System divides all questions as interrogative questions and rhetorical questions (Rodney P. Riegle. 1976) It could be used to improve teachers questioning techniques by making them aware of, and thus getting them to avoid, questions that are semantically ambiguous. In addition, this scheme also could be better equipped to handle students semantically ambiguous questions.

Using the Research for Effective Supervision: Measuring a Teacher's Questioning Techniques ( Roberts & Zody, 1989) The assessment tool designed for measuring teachers questioning techniques in this study is propped up by various studies, which have strong reliability and validity. Based on those studies contentions about effective questioning, this research summarizes that the assessment for effective questioning should include: Teaching Style, Level and Type of Questions, Wait time, Modeling of Mental Processes Used to Arrive at An Answer, as well as Transfer and Systematic Questioning (Vogler, 2005; Roberts & Zody, 1989). Jo Roberts and Melody Zody (1989) develops as an instrument the Measure of Effective Questioning Techniques (MEQT), which aims to help the supervisor and teacher focus on the teachers behaviors and students initial responses in key questioning areas. This instrument encourages teachers to employ three types of interactive (questioning) teaching styles, create varied cognitive levels of questions, be patient for the appropriate wait time after questioning, model mental processes by asking probing questions, and utilize systematic questioning for transferring, while supervisors should seek to observe and tally teachers specific behaviors or relative data during their questioning.
Page 5 / 18

Pre-service Teachers Thoughts about Teachers Questions in Effective Teaching Process (Cakmak, 2009) Cakmak (2009) employed surveys as a research approach to investigate pre-service teachers perceptions about teachers questioning in effective teaching contexts. The participants were provided with both closed-ended and open-ended questions. Data based on closed questions was analyzed statistically whereas data based on openended question was analyzed through content analyses. The findings of the study indicated that pre-service teachers valued evaluation of learning as an important facility of questioning; stated subject knowledge affected the questioning the most; intended to use questioning as a strategy to motivate student to learn.

Purpose of Using Literature for research

Roberts & Zody (1989), and Riegle (1976) indicated that when designing assessment of teachers questioning effectiveness, challenges such as accessibility, observability and measurability must be addressed. The article Classifying Classroom Questions (Riegle, 1976) provides the most popular question classification as six types of thinking, which can be utilized for identifying the frequency of faculty use and the preference of both faculty and students. The article Using the Research for Effective Supervision: Measuring a Teacher's Questioning Techniques (Roberts & Zody, 1989), provides an optional and observable measurement instrument, which also can be used for identifying the difference between faculty and students perception about effective questioning in terms of influencing factors. The last article, Pre-service Teachers Thoughts about Teachers Questions in Effective Teaching Process (Cakmak, 2009), offers two valid questionnaires, especially useful is the sufficient responses to the open-ended questions, which can be modified as closedend questions for further study.
Page 6 / 18

Methods
Research Design Since the present study aims to identify the relationship between faculty and students perceptions of effective questioning, an analytical study is used as the main research approach. Based on the fact that a survey allows for the collection of sufficient data easily from a large body of respondents quickly, two online surveys designed for faculty and students are respectively administered in this study. The questions are cited and reformed as closed-ended questions from the three articles stated above, which provide a quantitative way of collecting data. Despite the difference in the section of demographics, other items which focus on perceptions of questioning effectiveness and evaluation of facultys questioning effectiveness are basically the same. The topic of each question is listed below: 1. The factors that contribute to questioning effectiveness 2. The order of the factors which affect teachers effective questioning 3. The reasons for teachers using effective questioning 4. The types of teaching styles that provide the most effective questioning 5. The preference for the types of questioning that teachers use in class 6. The reasons for preference of the types of questions chosen 7. The most common teaching style among IS faculty regarding questioning techniques 8. The assessment of effectiveness of IS facultys questioning 9. The frequency of the six types of questions that the IS faculty uses

Participants The population altogether was 31 participants. A total of 6 faculty members and 25 students of the IS program at FSU participated in this study. Although the sample size is small, this study still can provide a perspective of the different perceptions about questioning effectiveness between faculty and students, since current literature rarely discusses this aspect.

Page 7 / 18

Research Instruments The data was collected through the questionnaire that consisted of 16 items for faculty and 17 items for students (the items of demographic for faculty is one item less than the students). The closed items were divided into three sections regarding demographics, perception of questioning effectiveness, and the evaluation of IS facultys questioning effectiveness. A rank order item was used in the questionnaire to examine the factors which affect teachers effective questioning in terms of faculty and students preference. The questionnaire also included three like-scales demo. The first one was used specifically for examining the facultys questioning effectiveness in light of the aspects as clarity, accuracy, inspiring thinking, transfer, motivation, and depth of follow-up questions. The second one was used for identifying the frequency of six types of questions for faculty use. The last one was used for generally assessing the facultys questioning effectiveness.

The items in the questionnaire were all based on the related literature stated above. In addition, one instructors (Dr. Shute) suggestions were also taken into account, who suggested modifying and reorganizing the questions in light of enhancing the present studys validity and reliability. A pilot study was undertaken with three IS program students after finalizing the questionnaire. Next, with the instructors (Dr. Shute) approval, two online were respectively administrated to all faculty and students of IS program. All the data were collected during March 25th to April 7th, 2012.

Data Analysis

Based on the nature of the closed-end questions, data from this study were analyzed in terms of percentages, means and standard deviations, which were calculated for the responses to each item. In addition, the data respectively collected from both surveys for faculty and students were further compared in terms of two sections: perception of questioning effectiveness, and evaluation of IS faculty questioning effectiveness.

Page 8 / 18

Findings and Discussion

Perception of Questioning Effectiveness 1. The factors that contribute to questioning effectiveness What makes questioning effective?
120% 100% 80% 60% 40% 20% 0% Clearness & Inspiring easily thinking understood Transfer Motivation Follow-up Questions Other factors 17% 4% 100% 92% 67% 60% 83% 72% 52% 33% Faculty Students 83% 68%

Other factors: Faculty-(1)Elements of surprise to capture attention (2)Causing students to really think about an idea or process Students-- Relating the question to something already known

Figure1: Factors that contribute to questioning effectiveness

100% of faculty and 92% of students strongly agree that clarity is the most important attribute for making effective questions, while both consider inspiring thinking less important. However, the figure also indicates that there are two prominent conflicts of perceptions between two populations. 52% of the students state that effective questioning should involve transferring knowledge, while only 17% of faculty agrees with them. On the other hand, 33% of faculty states that other factors, such as elements of surprise to capture attention, or causing students to focus on the idea, may contribute to making effective questioning as well, while only 4 % of the students think so.
Page 9 / 18

The findings here indicate that students and faculty both have commonalities in formulating effective questions in light of four attributes, such as clarity, inspiring, motivation and developing the follow-up questions. However, students are more concerned with whether a question can help them transfer the knowledge into a new setting for problem solving. Therefore, the IS program faculty may consider this factor while selecting or creating questions.

2. The order of the factors which affect teachers effective questioning

The order of factors which affect teachers effective questioning Rank Highest Lowest

1
Faculty Students 30% 4% Faculty 0% 17% 0% 0%

2
Students 22% 30% Faculty 50% 33%

3
Students 30% 9% Faculty 33% 33%

4
Students 9% 17% Faculty 17% 17%

5
Students 13% 39% Faculty 0% 0%

6
Students 4% 9%

Students attention Students knowledg e level Teachers subject knowledg e Teachers questioni ng skills subject characteri stics evaluating needs

33%

30%

33%

22%

17%

39%

17%

9%

0%

9%

0%

0%

50%

30%

33%

26%

0%

13%

17%

30%

0%

4%

0%

4%

0% 17%

9% 4%

17% 0%

4% 4%

0% 0%

9% 9%

0% 0%

13% 30%

33% 33%

39% 4%

50% 50%

35% 57%

Table 2: The order of the factors which affect teachers effective questioning

Overall, Table 2 shows the various orders of the influencing factors which are respectively ranked by the faculty and the students. The results indicate that three influencing factors of effective questioning are ranked the highest by students: students attention, teachers subject knowledge and teachers questioning skills . However, the faculty states that teachers questioning skills affect the questioning effectiveness the most. As to the lowest level, 57% of the students referred to the needs of evaluation, whereas the faculty ranks the subject characteristics and evaluating needs equally.

Page 10 / 18

The findings above reveal the different order ranked by faculty and students may be due to their roles in daily instruction. Since 67% of IS faculty has more than 10 years of teaching experience, especially the research experience, they may be more aware of the questioning skills, which help them deliver their proficient subject knowledge to the students. Also, based on the career responsibilities, the faculty has to take in to account evaluation and the subject characteristics, while students generally dislike being evaluated.

3. The reasons for teachers using effective questioning

Why should a teacher use effective questioning?

90% 80% 70% 60% 50% 40% 30% 20% 10% 0% motivation evaluation managing classroom critical thinking identifying prior knowledge 0% 12% 67% 50% 40% 50% Faculty Students 80% 72% 67% 68%

Figure3: The reasons for teachers using effective questioning

The reasons for teachers using effective questioning are very different in light of the faculty and the students, despite the reason identifying prior knowledge. The most important reason that 80% of the students value is motivation, which is only valued by 67% of faculty, 72% of students also assume effective question should inspire critical thinking, whereas only 50% of faculty agree with them, 12% of students also took in account managing classroom as a reason for using effective questioning, while no
Page 11 / 18

faculty considers it at all. In addition, 50% of faculty considers evaluation as one of the reasons for using effective questioning.

Since the participants are graduate students and university faculty, the faculty might assume their questioning should not be used for managing class, but for other aspects like motivation, evaluation, etc. In addition, students assumed teachers should use questions to enhance motivation and critical thinking abilities, which might engage them to learn effectively.

4. The types of teaching styles which provide the most effective questioning Which teaching style can provide the most effective questioning?
120% 100% 100% 80% 64% 60% 40% 24% 20% 0% 0% Directed Supervisory Interactive 0% 12% Faculty Students

Figure4: The types of teaching styles which provide the most effective questioning

All faculty members agrees that an interactive teaching style can provide the most effective questioning, while only 64% of students prefer this teaching style. Therefore, 24% and 12% of students assume directed teaching style and supervisory teaching style can provide the most effective questioning. This indicates that the other two

Page 12 / 18

teaching styles may meet the students needs, like fitting their own learning styles. The reasons will be inferred from the next item.

5. The preference for the types of questioning that teachers use in class
Faculty Students 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Knowledge Comprehension Application Analysis Synthesis Evaluation 0% 36% 33% 56% 50% 67% 64% 67% 60% 52% 83% 80%

Which type of questioning do you prefer using in class?

Figure5: The preference for the types of questioning that teachers use in class

Figure 5 indicates there are only two conflicts existing in the preferences of the types of questioning that teachers use in class. No faculty prefers using the questioning for merely recalling the knowledge, while 36% of students state they prefer this. In addition, the students report 19% higher rate of using questioning for evaluation than the faculty.

The findings here imply that the students not only prefer the questioning for evoking their relative lower thinking skills, but also the comprehensive or higher thinking skills as well, which both are beneficial for them to consolidate the prior knowledge, as well as for learning and applying the new knowledge and skills. On the other hand, the faculty may have higher expectations for their students. Therefore, they might prefer using other types of questioning for fostering students various higher thinking abilities.
Page 13 / 18

6. The reasons for preference of the types of questions chosen

Why do you prefer using the types of questions you chose above
90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 83% 64% 56% Faculty 17% 0% learning style questions on higher thinking test level learn effectively 20% 0% 4% Students 83% 80%

Other reason

Other reasons: Students--helps recall

Figure6: The reasons for preference of the types of questions chosen

56% of students state the questions should meet their learning styles, while only 17% of faculty agrees with that; 20% of the students indicate the questions should serve the tests, but no faculty prefers using this type of questions. The faculty would rather use the questions for evoking students higher thinking levels and engaging students learning effectively, instead of helping students cram for the exams. The data indicates that the faculty may have difficulties meeting all students learning styles. However, they all are willing to use various types of questions to help students develop their thinking levels, while refusing to use questions to serve the tests.

Evaluation of the IS facultys Questioning Effectiveness 1. The most common teaching style among IS faculty regarding questioning techniques

Page 14 / 18

Which tends to be the most common teaching style among IS faculty regarding questioning techniques?
60% 50% 50% 40% 33% 32% 30% 20% 10% 0% Directed Supervisory Interactive 17% 36% 32%

Faculty Students

Figure7: The most common teaching style among IS faculty regarding questioning techniques

50% of the faculty states that they intend to use an interactive teaching style in the daily instruction, however, only 32% of the students agree with it. In addition, 36% of the students indicate the faculty actually uses a supervisory teaching style, while 17% of the faculty claims to use this teaching style.

The reasons for the faculty and students having different perceptions of whether the teachers actually use the interactive instruction may be due to their different comprehensions of how interactive instruction is defined. And this is also the limitation of present study.

Page 15 / 18

2. The frequency of the six types of questions that the IS faculty uses
The frequency of each type of question that IS faculty use Min Value Max Value Average Value Faculty Students Faculty Students Faculty Students 2 2 5 5 2.83 3.48 3 3 3 3 2 2 3 2 2 2 5 5 4 4 5 5 5 5 5 5 3.83 4 3.67 3.67 3.67 3.96 4.2 3.64 3.68 3.52

Degree Knowledge Comprehension Application Analysis Synthesis Evaluation

Standard Deviation Faculty Students 1.17 0.82 0.75 0.63 0.52 0.52 1.21 0.73 0.71 0.7 0.69 0.77

Table8: The frequency of the six types of questions that the IS faculty uses

The standard deviations shown in Table 8 indicate that the faculty believes that they use questions for recalling knowledge were the least, while the questions used for the application are used the most. From the perspective of the students, who consider all types of questions are used by faculty are almost equally. The frequency of using application questioning types may indicate the IS faculty members higher expectations on graduate students, who should have solid foundation of lower thinking skills, and should focus on cultivate their higher thinking skills in the class. 3. The assessment of effectiveness of IS facultys questioning
Degree Clearness Accuracy Inspires thinking Transfers knowledge Motivates response Depth of Follow-up questions appraise the effectiveness of IS facultys questioning Min Value Max Value Average Value Faculty Students Faculty Students Faculty Students 3.5 1.9 4.5 5 4.05 4 4 2.3 5 5 4.45 4.2 3.6 3.2 2.8 3.2 2.1 2.2 2.2 2.6 4.5 4.6 4.6 4.5 5 5 5 4.5 4 3.93 3.9 3.97 4 3.89 3.97 3.76 Standard Deviation Faculty Students 0.33 0.77 0.33 0.7 0.29 0.53 0.74 0.43 0.76 0.76 0.74 0.58

Table9: The assessment of effectiveness of IS facultys questioning

Page 16 / 18

The standard deviations shown in Table 9 indicate that faculty members and students perceptions on the effectiveness of the questions are very close. They generally agree on the highest questioning effectiveness in terms of accuracy, while disagree on the depth of follow-up questions.

Conclusion
Several conclusions can be derived from the main results of the study as follows: There is a conflict between faculty and students perception of effective questioning. Students assumed that transferring knowledge could also make questioning effective, while faculty stressed the clarity, motivation and developing follow-up questions. Students also assumed that effective questioning could be used for motivation, inspiring critical thinking and managing class, while faculty focused on motivation and evaluation. Both groups agreed that effective questioning could be used for identifying prior knowledge. On the other hand, students assumed the most influential factors of effective questioning included students attention, teachers subject knowledge and teachers questioning skills, while faculty stated the teachers questioning skills impacted the effectiveness of questioning the most.

The faculty all preferred using the interactive teaching style, while only 64% of students preferred this teaching style. However, only 32% of students indicated the faculty actually used it. Faculty also dislikes using the questioning for merely asking to recall the knowledge, while students preferred those questions as well as the questions used for evaluation. Overall, the IS facultys questioning are highly effective, especially demonstrated in the aspects of clarity, accuracy and inspiring higher thinking, which are acknowledged by students as well.

Page 17 / 18

Reference
AKMAK, M. (2009). Pre-service teachers thoughts about teachers questions in. Elementary Education Online, 8(3), 666-675. JO Roberts & Melody Zody. (1989) Using the Research for Effective Supervision: Measuring a Teacher's Questioning Techniques. NASSP Bulletin, 73, 8-14 Rodney P. Riegle. (1976). Classifying Classroom Questions. Journal of Teacher Education, 27, 156-161 Vogler, K. E. (2005). Improve Your Verbal Questioning. The Clearing House, 79(2), 98103.

Page 18 / 18