Qualitative Research

Data Collection and Analysis

Purpose
Why qualitative research?
need for complex, detailed understanding of students experiences follow up to quantitative research further insight and explanation into relationships between variables

Characteristics of qualitative research project:

Researcher as instrument Inductive data analysis Participants meaning Interpretive inquiry
(Creswell, 2007)

Studio-Based Learning

Interview Protocol

1. What were your expectations for the course? Have they been met? Why/why not? 2. Did the course keep you interested and motivated to learn? Why/why not? 3. Did you find the process of completing programming projects helpful to you in learning about computer programming? Why or why not? 4. Has it been easy in this course to learn about other students programming projects? If so, have you found it helpful to learn about other students work? Why or why not? 5. Did you receive helpful feedback from the reviews of your programming projects by others? If so, was it from your TA or from reviews by other students? How, specifically, did it help you? 6. Did you find the process of writing and sharing reviews of others work helpful to you in learning computer programming? Why or why not? 7. Did you find the process of giving responses in the lab to others comments on your studio presentation helpful to you in learning computer programming? Why or why not? 8. Did you receive helpful feedback during your presentations in the lab? If so, was it from your TA or from students? How, specifically, did it help you?

9. Did you find the process of participating as an audience member in labs where other students presented their work and responded to comments about it helpful to you in learning computer programming? Why or why not?
10.Did you provide feedback during other students presentations? If not, why not?

Studio-Based Learning

Interview Protocol

11. Have you discussed course topics with other students or gone to them for help in this course? How about the instructor or TA? How often? When/Why? 12. Are you comfortable giving and receiving feedback on computer programming? Are you confident about the quality of the feedback you receive as well as the feedback you give?

13. Do you think most students in the course have had similar experiences to yours in working on programming projects in this course? Do you think that the programming projects and labs provided a learning experience that is shared by all students in the class?
14. In your view, what impacted your learning more: the lectures, the studio sessions or the jGRASP environment? 15. Has taking this course made you feel more comfortable with computer programming? With interacting with others about problems and solutions? 16. Has the course increased or decreased your interest in computer science or computer/software/wireless engineering? What parts of the course increased your interest? What parts of the course decreased your interest? 17. Will you recommend this course to other students? Why or why not? 18. Did you like working in pairs? Why or why not? If not, do you prefer working alone or in a group of different size? Please explain.. 19. Separate from the issue of working in pairs, Is there anything that you would change about the design of the course? Please explain. 20. What are your plans for taking computer science or computer/software/wireless engineering courses in the future? Do you plan to major in computer science or computer/software/wireless engineering?

Traditional

Interview Protocol

1. What were your expectations for the course? Have they been met? Why/why not? 2. Did the course keep you interested and motivated to learn? Why/why not? 3. Did you find the process of completing programming projects helpful to you in learning about computer programming? Why or why not? 4. Has it been easy in this course to learn about other students programming projects? If so, have you found it helpful to learn about other students work? Why or why not? 5. Did you discuss programming projects with other students and provide feedback to them? If so, what form did that feedback take (written, verbal)? If not, why not? 6. Were solutions to programming projects made available? Were these discussed in class? Did you find the availability and/or discussion helpful to you in learning the course topics? Why or why not? 7. Did you receive helpful feedback on your programming projects? If so, from whom did you get such feedback? How, specifically, did it help you? 8. Have you discussed course topics with other students or gone to them for help in this course? How about the instructor or TA? How often? When? Why? 9. Are you comfortable giving and receiving feedback on computer programming? Are you confident about the quality of the feedback you receive as well as the feedback you give?

Traditional

Interview Protocol

10. Do you think most students in the course have had similar experiences to yours in working on programming projects in this course? Do you think that the programming projects and labs provided a learning experience that is shared by all students in the class? 11. In your view, what impacted your learning more: the lectures, the studio sessions or the jGRASP environment?

12. Has taking this course made you feel more comfortable with computer programming? With interacting with others about problems and solutions?
13. Has the course increased or decreased your interest in computer science or computer/software/wireless engineering? What parts of the course increased your interest? What parts of the course decreased your interest?

14. Will you recommend this course to other students? Why or why not?
15. Is there anything that you would change about the design of the course? Please explain. 16. What are your plans for taking computer science or computer/software/wireless engineering courses in the future? Do you plan to major in computer science or computer/software/wireless engineering?

Interview Participants
Studio-Based Learning Course Participants
5 10 10 7 0

Semester/ Year
Fall 2007 Fall 2008 Spring 2008 Fall 2009 Spring 2009

Traditional Course Participants

5 0 0 8 7

Spring 2010
Total

5
37

5
25

Coding Process

(Creswell, 2007)

Coding Process
Data Analysis
organize data into file folders, index cards, or computer files convert files to appropriate text units for analysis either by hand or computer

The data generated by qualitative methods are voluminous. I have found no way of preparing students for the sheer massive volumes of information with which they will find themselves confronted when data collection has ended. Sitting down to make sense out of pages of interviews and whole files of field notes can be overwhelming.
(Patton, 1980)

Coding Process
Data Analysis
get a sense of the whole database write memos in the margins of transcripts

Coding Process
Data Analysis
describe in detail develop themes through some classification system provide an interpretation in light of personal views or perspectives from literature
independent coding

whole group discussion

In short, no absolute rules exist except perhaps this: Do your very best with your full intellect to fairly represent the data and communicate what the data reveal given the purpose of the study.
(Patton, 2002)

pair coding

whole group discussion

Data Analysis
In short, no absolute rules exist except perhaps this: Do your very best with your full intellect to fairly represent the data and communicate what the data reveal given the purpose of the study.
(Patton, 2002)

Coding Process

Data Analysis
In short, no absolute rules exist except perhaps this: Do your very best with your full intellect to fairly represent the data and communicate what the data reveal given the purpose of the study.
(Patton, 2002)

Coding Process

Data Analysis
In short, no absolute rules exist except perhaps this: Do your very best with your full intellect to fairly represent the data and communicate what the data reveal given the purpose of the study.
(Patton, 2002)

Coding Process

Coding Process
Data Analysis
present the data in a text, tabular, or figure form develop hypotheses or propositions that specify the relationship among categories of information

Codebook v. 8 Atlas.ti

Coding Process
Studio Based Learning vs. Traditional

SelfEfficacy

Context

Learning

Motivation

Engagement

Codebook Structure
Primary Themes

Context

Learning

Motivation

SelfEfficacy

Engagement

Qualifiers
Qualifiers
Qualifier: Affective Response to

Codebook Structure

A qualifier may be attached to a code as a label and helps determine the interpretation. Any of the following qualifiers can be attached to learning context, learning, motivation, engagement, and self-efficacy. explicit statement of expressed I really enjoyed it, so.. feelings and attitude (only applies to context and learning)
explicit statement of
I basically just expected to learn some advanced problem solving techniques.

Qualifier: expectation before Expectations of experiencing the course explicit statement of judgment Qualifier: with respect to degree of Evidence of excellence or effectiveness quality Qualifier: Evidence of learning from

I am confident about the quality of the feedback that I give, Im about 50/50 on the quality of the feedback I receive.

explicit statement of knowledge Did you find the process of writing and sharing reviews of others work helpful to gained from course experience you in learning computer programming? Its interesting because get to learn other approaches and learned from other (only applies to context)
approaches.

Only code positive/negative/ mixed where explicit

Positive/ Negative/ Mixed/ Neutral

Positive: Ok, did you find the process of completing programming projects helpful to you in learning computer programming? Yes, all of the projects that we had were targeted at us using a specific new data structure we just learned, or a new something, so with each project I did learn efficiently how to use a new something. Negative: Shamefully I didnt always pay attention in the lectures, otherwise the lectures would have impacted me more Mixed: Did you find that helpful, to learn about other students work? Yeah, it provided alternative ideas besides the ideas that were in our own group. Sometimes they were better, sometimes they were worse.

Definitions/Examples
Learning Process
Revised Blooms Taxonomy A. B. C. D. E. F. Remember Understand Apply Analyze Evaluate Create

Codebook Structure
A. NO EXAMPLES FOUND (Possibly not applicable) B. You get another person to give you an idea, so potentially you can get new and better ideas to compare yours against. C. Yes, it was very nice because one of the interesting things Ive noticed is that for every single problem that you try to solve with a program, theres any number of ways to go about solving it. And you know, some of them are more efficient time wise, some of them are more efficient memory wise, some of them are just not efficient at all, but, theres just any number of possible solutions. D. Ive had previous experience with except for some of the algorithms and I guess the inner workings, like Ive used the data structures that were using before. But Ive never really, you know, it was just like a black box. You know, I put stuff in and I get stuff out. But now I have a better understanding of, you know, how those really work and whats going on behind the scenes. E. The grading of another groups submission ended up being very important for solidifying the course material. That one aspect, of actually grading, with all of the weight and responsibility of the instructor, you know, as far as trimming the points andWell, if forces the grader to basically go back over the material that the stuff is covering and to solidify it in their own mind and be sure that they know what they are talking about if they are going to give some critical input on what somebody else has done. F. Well, for me learning is easier if Im trying to do something. If Im trying to solve a problem, its easier for me to come up with a solution than to have someone tell me this is the solution, remember it. Im more likely to remember how to do something if Ive done it than if Im just told how to do it.

The way in which someone learns

The cognitive process dimension of the Revised Blooms Taxonomy (RBT) is constructed as verbs used to define the process of learning. Make the following assumptions when coding: 1. The categories are not structured as a cumulative hierarchy, meaning that one category does not have to build from the mastery of another category. 2. Code ALL evident processes of learning . A. Retrieving, recognizing, and recalling relevant knowledge from long-term memory B. Constructing meaning from oral, written, and graphic messages through interpreting, exemplifying, classifying, summarizing, inferring, comparing, and explaining C. Carrying out or using a procedure through executing, or implementing D. Breaking material into constituent parts, determining how the parts relate to one another and to an overall structure or purpose through differentiating, organizing, and attributing E. Making judgments based on criteria and standards through checking and critiquing F. Putting elements together to form a coherent or functional whole; reorganizing elements into a new pattern or structure through generating, planning, or producing (Anderson & Krathwohl, 2001, pp. 67-68)

Examples

Codebook Structure Unit of Analysis

Interviewer: In your view, what impacted your learning more, the lectures, studio sessions, or jGRASP environment? Interviewee: I guess the studios. Shamefully I didnt always pay attention in the lectures, otherwise the lectures would have impacted me more, but I think actually listening to peoples [Qualifier-Evidence of learning from] [Studio] [Global-Positive] [Global-Negative]Structure-Lecture] [Context-Course [Studio] [Global-Negative] [Context-Course Structure-Lecture] [Engagement-Other] [Context-Course Structure-StudioPresentation] [Engagement-Other] learning from] [Qualifier-Evidence of [Context-Course Structure-StudioPresentation] [Global-Positive]

designs in studio and programming with my

partners was the most learning I got.

(Excerpt from studio based learning course interview)

Codebook Structure Multiple Coding

Interviewer: Did you receive helpful feedback from the reviews of your programming projects [Qualifier-Evidence of quality] [Global-Positive] by others? [Q06s-Helpful peer reviews] Interviewee: Yeah, um, it was good like wed

talk about okay heres an idea and people

would throw out like have you thought about doing it like this? Or, how exactly would you implement it? Or just help us think through okay like this is how were thinking about it but is actually practical? So, definitely, the feedback was really good.
(Excerpts from studio based learning course interview)

[Context-Course Structure-Assignment] [Context-Course Structure-Studio-Peer Review] [Context-Interaction-Receiving peer feedback]

[Learning-Process-Revised Blooms Taxonomy (RBT)-Apply] [Learning-Process-Revised Blooms Taxonomy (RBT)-Understand]

Codes/Subcodes
Interaction
Interaction my include helping peers learn, providing/receiving/responding to peer feedback, TA/instructor feedback Distinct classroom elements which together compose the structure of a course. These elements can include organization (lecture, labs, studio, programming environment) and activities (assignment). Some of these elements may be present only in a studio or traditional implementation. Quantifying attributes of a course

Context

Course Structure
Course Descriptors

Instructor Characteristics

Attributes of an instructor or teaching assistant The way in which someone learns Subjects or topics covered in the curriculum of the course The demonstration of proficiency in performing an action Preference for method of interacting with material for learning Motivation determined by the expectations of success and the value of success (i.e. grades, job utility, graduation, usable skills in the real world) Motivation determined by the inherent interest in an activity and/or knowledge gained The degree of participation and attention given toward an activity The feeling of competence to effectively perform a specific task

Learning
Motivation

Process Content Skills Style Task Value Interest Engagement Self-Efficacy

Coding Example 1
Studio Based Learning Course Interviewer: What were your expectations in 2210? Interviewee: Um, my expectations were pretty much uh continuous learning of the, I guess, basic constructs used in computer science. You know, different types of objects that we use to represent data. Um and an

introduction to some of the very basic algorithms that we use to deal with those objects.
Interviewer: And have your expectations been met? Interviewee: Yeah, Id Id say they have. Umyeah. Interviewer: Okay. Did the course keep you interested and motivated to learn?

Interviewee: Definitely [laughs]

Interviewer: Tell me. Can you explain that more? Interviewee: Um, the projects. Uh, it it really was the studio section I think that drove me more than anything. The..most of the stuff that were learning in the classroom Ive had previous experience with except for some of the algorithms and I guess the inner workings, like Ive used the data structures that were using before. But Ive never really, you know, it was just like a black box. You know, I put stuff in and I get stuff out. But now I have a better understanding of, you know, how those really work and whats going on behind the scenes.

Coding Example 2
Traditional Course Interviewer: Okay. Has it been easy in the course to learn about other students assignments and programming..programming projects? Interviewee: Um, notprobably not so much because this has been the traditional approach and we

havent really seen or interacted with other students. Weve all been told to do our work individually. Um, I
mean I guess in the little bit Ive spoken with other students um it seems so, but uh I couldnt, probably couldnt give you a great answer with that. Interviewer: Did you discuss assignments and programming projects with other students and provide

feedback to them?
Interviewee: Um, on very rare occasions. Typically I I do a lot of my work on my own and I like to challenge myself to see if I can get it.. Interviewer: Okay, thats

Interviewee: um, instead of asking for help. But but then again, I would uh I would also shhhyou
know if we were working in the lab..if I was working in the computer lab and theres some other people in there too, they would either say something or I would..they would ask something and I would try to bounce it off them. So that did happen on on occasion.

Coding Example 3
Studio Based Learning Course Interviewer: Um, has it been easy in this course to learn about other students group projects? Interviewee: Uh, yes actually actuallywas the uh the peer review process I I I think worked out fairly well. Interviewer: Did you find the process of writing and sharing reviews of others work helpful to you in learning algorithms? Interviewee: Yes, maam. Interviewer: How? Interviewee: Uh, just reading the the different approaches that some, some other students take is helpful to have a second set of eyes on any given problem, so seeing where other students may have used a different strategy helped me to basically learn more strategies to solve the different..a a given problem.

Lessons Learn(ed)(ing)
Qualitative research demands extensive time and resources Limit interview questions

Invest in learning qualitative computer software programs before making decisions about coding structure Meet frequently
Form pairs or small groups then discuss as a whole group Inter-rater reliability method

Implications
Codebook directly applicable to CPATH II survey responses Codebook directly applicable to other computer science studio based learning courses and traditional courses Codebook indirectly applicable to other courses within the field of engineering Process applicable to developing different codebook