You are on page 1of 57

RESEARCH METHODS HANDBOOK

HARVARD LAW SCHOOL FALL 2010


Office of Clinical and Pro-Bono Programs Austin Hall, Rooms 102 & 108 T: 617.495.5202 F: 617.496.2636 www.law.harvard.edu/academic/clinical Negotiation & Mediation Clinical Program Pound Hall, Suite 513 T: 617.496.7109 F: 617.495.7818 www.law.harvard.edu/negotiation

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

TABLE OF CONTENTS
INTRODUCTION ..........................................................................................................1
A. B. What should you take away from this material? ........................................................................................1 How is this guide organized? ........................................................................................................................2

CRASH COURSE IN STATISTICS .................................................................................3


A. B. What is Statistics? .........................................................................................................................................3 Error...............................................................................................................................................................4
Measurement Error ................................................................................................................................................................ 4 Sampling Error....................................................................................................................................................................... 4

C.

Sampling.........................................................................................................................................................5
When to Sample...................................................................................................................................................................... 5 How to Sample ....................................................................................................................................................................... 5 Probability Samples ............................................................................................................................................................... 6 Random Numbers Table ......................................................................................................................................................... 6 Systematic Sampling............................................................................................................................................................... 7 Cluster Sampling.................................................................................................................................................................... 7 Stratified Sampling................................................................................................................................................................. 7

D. E.

Sample Size ....................................................................................................................................................8 Descriptive Summary Methods ....................................................................................................................9


Measures of Central Tendency............................................................................................................................................... 9 Measures of Dispersion........................................................................................................................................................ 10

F. G. H. I. J. K. L. M.

Wait a Moment Why am I Reading About Statistics ??? ...................................................................10 Probability ...................................................................................................................................................11 Normal Distribution ....................................................................................................................................11 Sampling Distributions ...............................................................................................................................12 Confidence Intervals ...................................................................................................................................12 Hypothesis Testing ......................................................................................................................................13 Regression ....................................................................................................................................................13 References ....................................................................................................................................................13

SURVEYS.................................................................................................................. 15
A. B. Getting Started ............................................................................................................................................15 Types of Surveys..........................................................................................................................................15
Mail Surveys......................................................................................................................................................................... 16 Telephone Surveys................................................................................................................................................................ 17 Face-to-Face Surveys........................................................................................................................................................... 18 Internet Surveys.................................................................................................................................................................... 19 Hybrid/Mixed Mode Surveys ................................................................................................................................................ 20

C.

Survey Errors ..............................................................................................................................................21


Coverage Error .................................................................................................................................................................... 21 Sampling Error..................................................................................................................................................................... 21 Measurement Error .............................................................................................................................................................. 21 Nonresponse Error............................................................................................................................................................... 21

D. E.

Sampling.......................................................................................................................................................22 Question writing ..........................................................................................................................................23


Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

Turning Ideas into Questions ............................................................................................................................................... 23 Attitudes & Beliefs vs. Behavior & Attributes ...................................................................................................................... 23

F.

Question Structure ......................................................................................................................................24


Open-ended Questions ......................................................................................................................................................... 24 Close-ended Questions......................................................................................................................................................... 25 Some Final Tips.................................................................................................................................................................... 26

G.

Questionnaire Design ..................................................................................................................................28


The First Question................................................................................................................................................................ 28 Question Order..................................................................................................................................................................... 28 Question Clusters ................................................................................................................................................................. 29 Narrative .............................................................................................................................................................................. 29 Oral vs. Written Surveys....................................................................................................................................................... 29

H.

Analyzing Results ........................................................................................................................................30


Cleaning and Coding ........................................................................................................................................................... 30 Analysis ................................................................................................................................................................................ 30 Interpretation ....................................................................................................................................................................... 31

I. J.

Ethics ............................................................................................................................................................31 References ....................................................................................................................................................31

INTERVIEWS ............................................................................................................ 33
A. General Guidelines ......................................................................................................................................33
Introduction.......................................................................................................................................................................... 33 Questions.............................................................................................................................................................................. 33 Closing ................................................................................................................................................................................. 34

B.

Recommended Questions............................................................................................................................34
General Questions................................................................................................................................................................ 34 Ongoing Conflict.................................................................................................................................................................. 35 No Specified Conflict............................................................................................................................................................ 35

C.

Positions to Interests ...................................................................................................................................36

FOCUS GROUPS ....................................................................................................... 38


A. B. C. D. Uses of Focus Groups ..................................................................................................................................38 Combining Focus Groups with Other Research Methods .......................................................................39 Strengths and Limitations of Focus Groups .............................................................................................40 Planning for the Focus Group ....................................................................................................................41
Choosing Participants.......................................................................................................................................................... 41 Number of Groups................................................................................................................................................................ 41 Heterogeneous vs. Homogeneous Groups............................................................................................................................ 42 Group Size............................................................................................................................................................................ 42

E.

Preparing the Focus Group ........................................................................................................................43


Recruiting Participants ........................................................................................................................................................ 43 Choosing the Location ......................................................................................................................................................... 43 Interview Guide.................................................................................................................................................................... 44

F. G.

A Word About Group Dynamics ...............................................................................................................44 Conducting the Focus Group .....................................................................................................................45
The Moderator ..................................................................................................................................................................... 45 How involved should the moderator be? .............................................................................................................................. 45 Getting Started ..................................................................................................................................................................... 47 Guiding the Discussion ........................................................................................................................................................ 47 Making Sure Everyone Participates ..................................................................................................................................... 47 Probing ................................................................................................................................................................................ 47 Managing Time .................................................................................................................................................................... 48 Dealing with Common Problems.......................................................................................................................................... 48

H.

Analysis ........................................................................................................................................................49
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

Qualitative Analysis ............................................................................................................................................................. 49 Content Analysis................................................................................................................................................................... 49

I. J.

Ethics ............................................................................................................................................................50 References ....................................................................................................................................................50

BIBLIOGRAPHY ....................................................................................................... 51 GLOSSARY ............................................................................................................... 52

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

HNMCP RESEARCH METHODS HANDBOOK

INTRODUCTION
Welcome to the Negotiation and Mediation Clinic! By now you should know what project youll be working on this semester, and youre probably thinking about how you should get started. But as you crack open this handbook, you might be wondering why we want you to learn about research methods. Well, here are some examples of how research methods can help you execute your project and produce solid results: A military college recently became co-ed, but is having trouble retaining its female students. Youd like to learn about the female experience at the college, but you dont even know what kinds of questions to ask. Using a focus group to bring together female students for a discussion could help you figure out why theyre leaving the school. The Catholic Church has found an increasing number of disputes arising among clergy members, and wants to develop a dispute resolution system to help manage them. Administering a survey to clergy members might help you figure out what the current system looks like, and how people within that system would like to see it changed. A large international NGO wants to streamline its decision-making procedures, which are currently ad hoc and decentralized. Conducting interviews of key players within the NGO might help you understand what the current procedures are and which changes are likely to be feasible.

The purpose of this handbook is to give you a brief overview of major research methods as they apply to projects for this clinic. Its not designed to function as an in-depth explanation of these methods, but as an introduction to get you started on the path to reliable research that you can present to your client. Remember, these are real clients with real problems that need real solutions, so we expect you to be responsible in your research and analysis, which might mean consulting additional resources. So with that said, happy researching!

A. What should you take away from this material?


Our intent in giving you this manual was to give you a very generalized understanding of bestpractice research methodology. To be clear: In the ideal world, the clinics research would itself be in line with those best practices, and our findings statistically impeccable. That said, we of course realize that for most of our projects, PhD-level statistical accuracy is difficult if not impossible to achieve given our clinics resources and given the time you have to complete a project.1 So fear not we are not expecting your projects to demonstrate gold standard research methodology!

This is of course leaving aside entirely the question of whether perfect research

design is even necessary given the analytical and think tank nature of most of our clinical projects.

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

HNMCP RESEARCH METHODS HANDBOOK

After reading this manual, you hopefully should be able to do three things: 1. Strategize about your own research methodology: You and your teammates should be able to strategize about how to maximize the rigor of your projects research, taking into consideration your groups limited capacity, time and resources available to carry out and analyze research. 2. Identify your projects research shortcomings: After reading this guide, you should be able to identify ways in which your projects research methodology deviates from the gold standard to which you would be held if you were a PhD candidate or professional statistician. In other words if a statistician or an econometrician were to carefully scrutinize your projects conclusions and research methods, what would she look for when judging whether or not our conclusions were accurate? 3. Speculate about the impact of your research methodologys shortcomings: Finally, you should be able to speculate about the impact of your justifiable research methodology shortcomings. If you collected data in a certain way and suspect that there might be an unavoidable error of some sort, how might your conclusions be different had you somehow been able to correct for that error? Most final projects will include a methodology section of some sort, in which you should specifically put into writing your thinking about the above three points.

B. How is this guide organized?


Given our purposes for subjecting you to this material, we structured the guide with the overview of statistics preceding the description of survey and focus group methodology. Our thinking was that a presentation of the GOAL of an activity should generally precede the description of HOW to achieve that goal. So, if you are the kind of person whose eyes glaze over when reading about statistics and research design, keep in mind that the first section covering statistics is the aspiration, which will hopefully make it clear why the methods we are prescribing will help you achieve good research outcomes.

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

HNMCP RESEARCH METHODS HANDBOOK

CRASH COURSE IN STATISTICS


Afraid of statistics? Breathe a sigh of relief. We dont expect you to be - or become, for that matter a statistics expert for this course. But since the research you do in this clinic is for real clients who have real stakes in your work, we want you to be able to identify how dependable your research is, where inaccuracies may lie, and how further analysis (including math!) could strengthen your work. This section will introduce the basics of statistics, but it is by no means designed to replace a rigorous statistics class. And dont worry - even though well introduce some key statistical concepts and methods, our goal is not to have you do complex regression analyses or hypothesis tests after reading the next few pages. Instead, we want to familiarize you with considerations you should take into account when sampling, analyzing, and reporting data. Weve listed a couple of additional resources at the end of this section, but also we recommend that you consult other resources, like Harvards economics department, if you feel that you need more guidance.

A. What is Statistics?
So what is statistics? Well, the basic idea is that its a way to learn about a population thats too big to study in its entirety say, all doctors in the United States or all lawyers in China by looking at a sample of that population. But first things first: before we go on, youll need a few key definitions to get you speaking the language of statistics: Population - a population includes all of the individuals in a group you want to study. Individuals dont have to be people; they could be organizations, laws, or even rocks (if youre a geologist)! Sample - a sample is any subset of a population. Parameter - a parameter is a characteristic of a population. For example, if your population were lawyers in China, one parameter would be the percentage that is environmental lawyers. Statistic - a statistic is a guess of a parameter

So, using these definitions, the general idea of statistics is that when you have a population thats too big to examine as a whole, you cant properly analyze its parameters (or characteristics). To get information about the parameters youre interested in, you can look at the characteristics of a more manageable and representative portion, or sample, of the population. Then you can use statistics to make educated guesses about the populations parameters based on that sample. In other words, statistics are educated guesses, based on samples, about population parameters.

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

HNMCP RESEARCH METHODS HANDBOOK

B. Error
All research methods have room for error and bias, and well discuss these more in the following sections. When doing statistics, though, there are two main errors to be aware of: measurement error and sampling error.

Measurement Error
Measurement error relates to the quality of the data you collect, and is a concern in any research method. With regard to statistics, the idea is that if you collect data from a sample in a way that produces biased or unclear results, it simply cant represent the population accurately. For example, lets say we asked a sample of Harvard students Do you agree that the dining hall is fantastic? A question like this primes respondents to think about the dining hall more positively than they otherwise would, so it would probably produce more yes answers than wed get if we asked the question more neutrally. This, in turn, would cause us to report, based on our sample, that most Harvard students find the dining hall fantastic which probably isnt true. This means that no matter what research method you use, you need to make sure your data is as reliable and clear as possible; otherwise, you wont be able to use it to accurately judge the larger population.

Sampling Error
When you do statistics, the very fact that youre examining only a sample of a population means that youll have some degree of inaccuracy in your research this is sampling error. For example, lets say youre trying to estimate how many of Chinas lawyers specialize in environmental law by looking at graduates of a few major law schools in China. If these law schools represent the population of Chinese lawyers well, they should give you a good idea of what percentage of the population specializes in environmental law. Even so, there is simply no way that the sample can give you a precise number. So no matter what you do, you can never avoid sampling error entirely. What you can do, however, is minimize sampling error by increasing your sample size. The other thing you can do is report it, which most professional researchers do. Youve probably noticed, for example, that when polls results appear in places like newspapers, theyre usually reported as accurate to plus or minus a few percentage points; those percentage points are the sampling error of the poll. While its fortunate that sampling error can be calculated, we dont expect you to calculate it for this class. We do, however, want you to be aware that its there and that its affecting the accuracy of your results.

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

HNMCP RESEARCH METHODS HANDBOOK

C. Sampling
As youve probably noticed by now, samples are the basis of statistics. Since the point of a sample (at least for statistics) is to generalize its results to an entire population, taking a good sample is fundamental to producing good statistics. In this section, well discuss how to sample and well describe a few of the most common sampling methods.

When to Sample
In almost all cases, you will have to sample. Only when you are analyzing a target group small enough so that you can accurately contact and question every single member of that population, or where your resources are so large that you can again contact and question every single member of a larger population, can you avoid sampling. In this case, you would be conducting a census. Think of the well known US Census, which is carried out every ten years. Estimates project that the 2010 census, which is intended by the Constitution only to enumerate the American population, will cost the US taxpayer $14.5 billion. Imagine the simple task of counting the number of people in the United States costs over 1600 times as much as the entire cost of constructing HLS Northwest corner! Unless your project is dealing with a small target population, or unless the subject matter of your clinical project is REALLY important, it is highly unlikely that HNMCP could ever muster sufficient resources to allow you to conduct a census.

How to Sample
There are three basic steps to sampling: 1. Identify the target population the first thing you need to do is decide who your target population is. In other words, whom are you studying? Is it the Chinese environmental lawyers? Students at HLS? Its helpful to have as precise a definition as possible. 2. Compile a sample frame a sample frame is a list of population members that youll use to draw your sample, like a phone book or a staff directory. Ideally, your sample frame should be complete, up to date and composed only of population members of interest. It should also list each population member only once. These characteristics will protect your research against errors, like choosing the same population member twice or accidentally making it impossible to select some population members. 3. Select the sample all samples fall into two broad categories: probability samples and nonprobability samples. Probability samples are ones chosen randomly, which allows them to be generalized to larger populations. Non-probability samples are ones whose members are chosen for specific reasons, like convenience or role in an organization. Because theyre chosen subjectively, non-probability sample results cant be generalized to the broader population, so we wont focus on them here. Instead, well review characteristics and methods of probability sampling.

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

6 Probability Samples

HNMCP RESEARCH METHODS HANDBOOK

A probability sample, sometimes called a random sample, is one where each member of the target population has an equal or known possibility of being selected for it. The randomness of this kind of sample, together with the laws of probability, make it likely that it will represent the population; this is what allows the researcher to generalize its results to the larger population with a calculated estimate of error. Its important to stress, however, that just because a particular sample is chosen randomly doesnt mean that its necessarily representative of the population. In fact, its quite possible for probability sampling to produce strange samples. For example, lets say youre studying Chinese environmental lawyers, and youre interested in where theyre from. Although the population as a whole may come disproportionately from Beijing, its entirely possible, although improbable, that you could draw a random sample that contains only lawyers from Shanghai, Guangzhou, and Hong Kong. The value of random sampling lies in probability at a broader level - its highly improbable that if you took ten or twenty more random samples of the Beijing-dominated population you would still end up with samples dominated by lawyers from other cities. There are several ways to take a probability sample, the simplest being drawing sample members from a hat. Well describe a few other common methods below:

Random Numbers Table


Random numbers tables are usually computer generated, although you can also find them in statistics textbooks and even online. They consist, as you might have guessed, of numbers generated randomly and ordered in a table format, as below:

39634 14595 61885

35050 95113 00020

38449 46574 86558

61303 46699 87474

15140 52925 03395

To illustrate this method, lets say you want a sample of 50 out of a population of 5000. The first thing you need to do is number the entire population from 1 to 5000. Then, start at the top of the random numbers table and work your way down the columns systematically, selecting any population member whose number appears until you have chosen 50 sample members. Using the above table, you would look at the first 4 digits of each number, since 5000 has 4 digits, choosing population member 3963, 1459, 3505, 0002, and so on.

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

HNMCP RESEARCH METHODS HANDBOOK Systematic Sampling

This method relies on the proportion of sample members to population members, using the proportion as an interval to select sample members. For example, if youre choosing a sample of 50 out of a population of 5000, youll be choosing 1 in 100 population members. To use this method, number all of the population members and then choose a starting point randomly. Then go down your population list choosing every 100th population member until you have 50 of them. Before you use this method, make sure that the population list doesnt break down into intervals that might bias your sample. For example, if youre dealing with days of the week and you choose every 7th population member, youll get the same day every time.

Cluster Sampling
This method breaks the population down into clusters most often geographic ones and then samples each cluster randomly. Its helpful for when a good, comprehensive list frame cant be easily compiled, since it allows researchers to construct lists only for clusters being sampled. To illustrate this method, lets say youre researching evictions by the Boston Housing Authority. You would break the city into clusters, like by neighborhood or voting ward, then randomly choose which clusters to sample, and then take random samples of the clusters chosen. Its important to point out that this method is problematic when each cluster is homogeneous within itself, since each one wont accurately represent the population as a whole.

Stratified Sampling
Researchers using this method divide the population into subgroups (or strata), and then take random samples of each one. This technique is helpful for when youre looking at a population as well as some of its subgroups, like if youre studying Chinese lawyers in general, but also want to learn about Chinese environmental lawyers in particular. When using this method, youll have to be aware of what percentage of the overall population each subgroup represents, since sometimes youll want samples to reflect their true representation in the population, but other times youll be more concerned with having samples of equal size. As a result, this technique can have complicated implications for both generalizing back to the overall population and for calculating sampling error.

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

HNMCP RESEARCH METHODS HANDBOOK

D. Sample Size
When youre getting ready to sample, one important thing to think about is how big your sample should be. Sample size depends on a few factors: Accuracy generally speaking, increasing sample size reduces sampling error, meaning that bigger samples are usually more accurate. You should consider your need for precision when deciding how big your sample should be. Variation populations that have a lot of variation in the characteristics youre studying usually require bigger samples. The idea here is that you want to be confident that any sample you choose would produce roughly the same results, which is unlikely to happen if youre taking small samples from a heterogeneous population. Subgroups - If youre looking at subgroups within a population, youll have to divide your sample into corresponding subgroups. In effect, this creates smaller samples of each subgroup, which increases the sampling error for each one. So keeping accuracy intact for subgroups means using a larger overall sample. Resources Finally, dont forget that youll probably have limited resources, especially time. Bigger samples require more resources, so youll have to find a balance between precision and efficiency.

This is just a quick summary of some important and complicated considerations. Wed recommend looking at the resources listed at the end of this section for more guidance.

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

HNMCP RESEARCH METHODS HANDBOOK

E. Descriptive Summary Methods


Once youve taken your sample, youll need to analyze it to figure out what it says about the population. Statistics uses a variety of measurement techniques, but two of the most fundamental ones are measures of central tendency and measures of dispersion. Well focus on these two types in this section, but we encourage you to do some outside research to learn about other methods.

Measures of Central Tendency


Measures of central tendency describe the middle or majority of the population. Theyre useful for statistics because we have a tendency to ascribe to an entire population what its average or middle group does. Just think, for instance, about how often you use an average to tell you whats normal for a population - like batting averages for baseball players or LSAT scores for HLS students. There are three main measures of central tendency: Mean : the mean (or average) is calculated by adding up all of the values in a group and then dividing that sum by the number of group members. Believe it or not, this is one of the most powerful tools in statistics! For this clinic, though, youll mostly look at the mean of a sample and use it to describe the corresponding population. The main problem with the mean is that outliers (numbers that are extreme or dont otherwise fit with the majority of the data set) throw off its calculation. unlike the mean, the median isnt a calculated value. Its simply the number in the middle of a data set, so you can find it by putting all data set numbers in ascending or descending order, and then picking the middle one. If you have an even amount of data set members, calculate the median by averaging the middle two. Although the median is not a particularly strong statistical tool, its advantage is that outliers do not affect it, so its a more stable measure of central tendency. the mode is the number that that occurs most commonly in a data set. Like the median, its not a calculated number and isnt a particularly strong statistical tool, but its a more stable measure of central tendency.

Median :

Mode :

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

10

HNMCP RESEARCH METHODS HANDBOOK

Measures of Dispersion
Dispersion looks at the range of data within a data set, so its most useful when you want to understand the variability in your data. For example, we would probably want to use some kind of dispersion to understand what different types of law Chinese lawyers are specializing in and in what numbers. There are several ways of measuring dispersion; well discuss a few of them below: Range : the range is simply the difference between the highest and lowest numbers in a data set. Since its not a calculated number, and since it depends only on the top and bottom extremes of a data set, its not a particularly strong statistical tool, but it does give you a sense of how much variation there is in your data.

Distributions : distributions are more helpful measures of dispersion, and are relatively simple to calculate. First, divide your data set into appropriate categories, and then count how many members of the data set fit into each one. So, for example, you might divide the Chinese lawyers into categories like environmental lawyers, business lawyers, immigration lawyers, etc. Then youd count up how many lawyers fit into each category to get your distribution. Its often most helpful to turn this type of distribution into a percent distribution, which tells you what percent of the population each category represents and thus provides a more readily comparable measure. Calculate the percentages by dividing the number of category members by the total number of data set members. Standard Deviation : standard deviation is the average difference between an individual data set member and the data sets mean. In other words, it tells you how values in a data set spread around the mean. This is one of the most common measures of dispersions, and is an extremely powerful statistical tool. Calculating standard deviation is fairly tedious, and since we dont expect you to do it for this course we wont go into an explanation here. The resources listed at the end of the chapter have both the formula for calculating standard deviation and more information about its uses.

F. Wait a Moment Why am I Reading About Statistics ???


Lets pause for a minute to sum up where we are so far: measures of central tendency and dispersion are some of the most useful and user-friendly ways of analyzing any data that youll be dealing with in this clinic. Combined with a good sample, these tools will help you present data thats fairly reliable and easy to understand to your clients. Youll also be able to recognize that, by virtue of using a sample, the accuracy of your research is limited. What we havent explained is why statistics works, or some of the methods statisticians use to explain more complicated issues like how variables affect each other how does being Chinese affect the likelihood that a lawyer will do environmental work? What follows is a brief summary of some of these topics, with the goal of familiarizing you with concepts of statistics and helping you see where more rigorous research and calculations could significantly hone your results.

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

HNMCP RESEARCH METHODS HANDBOOK

11

G. Probability
When we look at a random sample and then translate its results to a population, were relying on probability, defined as the relative frequency with which something is likely to happen. To illustrate the basis for this idea, think of flipping a coin. Theres an equal probability of getting heads or tails, or a probability of 0.5. But if you flip the coin twice, you wont necessarily get heads once and tails once. Instead, to really see heads and tails an equal number of times, youd probably have to flip the coin hundreds of times. This is the idea behind random sampling: if you took a whole lot of random samples, on average theyd represent the population accurately. This favorable probability allows you to make judgments about the population based on a single sample, even though you can never really know if that particular sample represents the population accurately.
Example: Imagine you were given the (incredibly boring) task of flipping a quarter 100 times, and counting up how many times you got heads and tails respectively. Q: how many times would you expect to get heads? The reason you (hopefully) answered both of the above questions with 50, is that we all know that chances of getting heads in a coin toss should be 50/50. If your clinical supervisor asked you to verify this assumption, and you had to conduct the 100 coin-flip exercise 100 times (using a fair coin) you would ultimately be able to chart your results as follows:
y-axis: number of times a 100-coin-flip exercise resulted in a result of x number of heads tosses. number of times you got heads as a result

H. Normal Distribution
The normal distribution is often called a bell curve its a distribution of data that looks like a bell, where the mean, median, and mode are all equal. The area underneath the curve represents probability.

x-axis: Normal distributions are fundamental to statistics because they relate 50 measures of central tendency and dispersion in a key way: the area underneath the curve that lies between one standard deviation in each direction from the mean encompasses 68% of all values in the data set. In other words, there is a 68% probability that any value in a normal distribution will fall within one standard deviation of the mean. Similarly, the area between two standard deviations represents 95% of all values, and the area between One standard three standard deviations represents 99% of all values. deviation

Statisticians calculate the numerical value of the standard deviation, and use it to indicate their certainty 50 in claiming that the findings of their representative sample is either 68%, 95%, or 99% certain to be within x of the findings had we surveyed the entire population (see confidence intervals below). Calculating the numerical value of the standard deviation requires calculations which can be found in any statistics book, but is most likely beyond the scope of a typical HNMCP clinical project.

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

12

HNMCP RESEARCH METHODS HANDBOOK

I. Sampling Distributions
Imagine you took a sample from a population, and then calculated its mean. Then imagine you took a new sample, and calculated that ones mean. Then imagine that you repeated this over and over again, until you decided to stop and graph all of the sample means you had calculated. What youd have would be a sampling distribution. Even though youd never do this in practice, sampling distributions are fundamental to statistics because they have a key property, which comes from the Central Limit Theorem: as long as youre working with a population with at least 600 members and samples with at least 30 members, then the sample means will be normally distributed around a mean that, on average, is equal to the population mean. This property of sampling distributions allows statisticians to do a lot of things, including making guesses about populations that arent normally distributed based on the characteristics of normal distributions. Thanks to the tireless work of statisticians who tediously proved the truth of the Central Limit Theorem, we can confidently present the findings we calculate from our sample mean, and estimate how likely it is that the true population figures are within x of our findings.
Example: Political Polling. Gallup is one of the most well-known international polling consulting firms. During the 2008 election for U.S. president, Gallup conducted daily surveys of 1000 or more registered U.S. voters. Using the methods described above, Gallup statisticians have estimated that this sample size will give them 95% accuracy within a +/- 3% confidence interval. If instead of 1000 voters Gallup interviews 2000 registered voters for a particular poll, their 95% accuracy confidence interval shrinks to +/- 2%.

J. Confidence Intervals

Even though sampling distributions let statisticians make strong assumptions about how a sample relates to Excerpt from August 28, 2008 poll tracking popularity of John McCain the population, they still can and Obama among registered US voters: never be sure that those assumptions are correct. Survey Methods Confidence intervals are a way For the Gallup Poll Daily tracking survey, Gallup is interviewing no fewer to manage but not eliminate than 1,000 U.S. adults nationwide each day during 2008. this uncertainty. For example, The general-election results are based on combined data from Aug. 25we cant be absolutely sure that 27, 2008. For results based on this sample of 2,723 registered voters, a sample mean is the same as the maximum margin of sampling error is 2 percentage points the population mean. We do, however, know that in a normal (http://www.gallup.com/poll/109897/Gallup-Daily-Obama-Moves-Aheaddistribution, 95% of all values 48-42.aspx) fall within two standard deviations from the mean, and that sample means are normally distributed around the true population mean. So we can construct an interval that is two standard deviations in either direction from the sample mean, and know that 95% of the time, or 95 times out of 100, the population mean will fall within that interval. Its important to highlight, however, that we still have no way of knowing if this particular samples confidence interval contains the population mean. That is, any particular sample could be one of Two the 5 times out of 100 that the population mean is not standard in the confidence interval. deviations

42% (of U.S. electorate support John McCain) (Aug. 25-27, 2008)

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

HNMCP RESEARCH METHODS HANDBOOK

13

K. Hypothesis Testing
Hypothesis testing is a way of using a sample distribution to test whether a hypothesis is likely to be correct, based on probability. Typically, researchers first create a null hypothesis to be either accepted or rejected, and then an alternative hypothesis, which states the opposite. For example, a null hypothesis might be that the population mean equals 50. The alternative hypothesis, then, would be that the population mean does not equal 50. Researchers then use their sample data to determine how probable it is that the null hypothesis is true; if it appears highly improbable that it is correct, researchers reject the null hypothesis, and if it appears probable they fail to reject the null hypothesis. Of course, since hypothesis testing is based on probability, and we can never be sure how well the sample reflects the population, it is entirely possible to reject a true null hypothesis (a type I error) or to fail to reject a false null hypothesis (a type II error).

L. Regression
Regression analysis is one of the more complex and useful topics in statistics. Essentially, it is a way to use statistics to explain reasons for variance in data. For example, if youre studying professors salaries and you notice that they vary a lot, you might use regression to understand how variables like gender, area of expertise, and number of articles published affect those salaries. The general idea of regression is that if you plot your data on a graph, usually with the independent variable on the x-axis and the dependent variable on the y-axis on a graph, you can draw a line (usually using a computer) through the data that is as close to as many points as possible. This is the regression line, and it gives you information about correlation, or how the two variables move together. It cant, however, tell you about causation; thats for the researcher to hypothesize.

M. References
1. Druckman, D. (2005). Doing Research: Methods of Inquiry for Conflict Analysis. Thousand Oaks, CA: Sage. 2. Newport, F., Saad, L. and Moore D. (1997) How are Polls Conducted? in Where America Stands 2d ed.. John Wiley & Sons, Inc. (available at Gallup.com website - http://media.gallup.com/PDF/FAQ/HowArePolls.pdf) 3. Salant, P. and Dillman, D. (1994). How to Conduct Your Own Survey. London: New York: Wiley. 4. Wright, S. (1979). Quantitative Methods and Statistics. London: Beverly Hills: Sage.

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

HNMCP RESEARCH METHODS HANDBOOK

15

SURVEYS
Surveys are everywhere these days! You cant stay in a hotel, take a class, or even use the internet without someone asking you to complete a survey. When internet pop-ups ask you to complete a survey about the website youre using, or when the U.S. Census Bureau sends you a profile to fill out, its because youre part of a population from whom these organizations want information. Their surveys are a systematic way of getting that information. The ubiquity of surveys, however, doesnt mean that theyre easy to administer. In fact, there is room for error at every step of the way, from choosing whom to survey to putting the survey questions in order. This section serves as an introduction to basic methods of designing, administering, and analyzing surveys. It is not intended as a comprehensive guide, however you should absolutely consult your professors throughout the survey process, and you may also need to consult the resources listed at the end of this section for further guidance.

A. Getting Started
The first thing you need to decide before doing a survey is what, specifically, you need to learn. Once you know what information youre looking for, you can begin designing the survey. For example, lets say youre helping a legal aid clinic find out why so many of their clients are being evicted by the local housing authority. One thing youll need to find out are the most common reasons for evictions are most people evicted for non-payment of rent? Noise complaints? Something else? Keep in mind that although you might like to know other information like the education level of most of those evicted; you may not need this information to assist the legal clinic. If youre not careful about limiting the amount of it would be nice to know information you seek, your survey will quickly become unwieldy. Once you have a sense of what you need to know, you can figure out who to ask and how to ask them.

B. Types of Surveys
In deciding what type of survey to use, youll need to consider the information you need, the characteristics of the population youre targeting, and your own resources (especially your time, given the length of academic semesters). For example, if youre pressed for time and want your survey results quickly, a phone survey might be the best option for you. On the other hand, if youve decided to target individuals who have been evicted from their homes, they are unlikely to have telephones and addresses; a face-to-face survey would be a better choice in that case. Below is a brief overview of survey methods, along with their major advantages and disadvantages:

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

16

HNMCP RESEARCH METHODS HANDBOOK

Mail Surveys
As you might imagine, researchers using this method send the survey to a target population or sample by mail, and then respondents mail the completed survey back to the researcher. Researchers typically get addresses from a list frame like a list of utility company customers or a list of employees from a human resources department, taking care to ensure that the list is as complete and accurate as possible. To improve response rates, researchers should send the survey with a cover letter that explains the purpose and importance of the survey and a pre-addressed and stamped envelope. It can also help to send an introductory letter before the survey and a reminder letter (or phone call) after the survey has been sent. Mail surveys are best for when you have limited resources, when you have an accurate address list, when a rapid response time is not required, and when the respondents have the necessary skills to respond accurately in writing. Their strengths and weaknesses are summarized in the chart below:

Mail Survey Strengths

Mail Survey Weaknesses

Low cost, infrastructure and staff needs Good control over how questions are asked High accessibility to most populations (most people have addresses) Gives respondents a sense of privacy

Long data collection time List frames often contain inaccuracies High non-response rates (think: junk mail!) Little control over who responds and in what order they respond to questions Difficult to use complex and open-ended questions, since researcher is not present to provide explanations

Enables the use of graphics and visual aids

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

HNMCP RESEARCH METHODS HANDBOOK

17

Telephone Surveys
Like mail surveys, phone surveys are conducted using an existing list frame, like the yellow pages or an organizational staff phone directory. The researcher then calls population or sample members using that list, and either administers the survey then or calls back at a more convenient time. (While various kinds of random dialing techniques also exist, you are not likely to use them for this clinic. If you find that you need to, however, we suggest that you consult an appropriate resource.) Unlike mail surveys, though, the surveyor and the respondents interact personally, meaning that you can introduce bias and inconsistencies if youre not careful to present the questions in the same way to everyone. Phone surveys are best for when you need to get responses quickly, when you have an accurate list frame, and when your target population is likely to have phones. Their strengths and weaknesses are summarized in the chart below:

Phone Survey Strengths

Phone Survey Weaknesses

Quick data collection time Good control over who responds to the survey, question order, etc. Low to moderate cost, depending on long distance calls and rates High accessibility to population (most people have phones)

List frames often contain inaccuracies. Interviewer can introduce bias if not careful

No visual communication

Certain populations are unlikely to have phones

Easy to use complex and open-ended questions

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

18

HNMCP RESEARCH METHODS HANDBOOK

Face-to-Face Surveys
Face-to-face surveys differ from interviews (discussed in the interview section below) in that surveyors administer the same questionnaire in person to a variety of respondents, as opposed to asking a series of questions tailored to the person being interviewed. The surveyor can either use a list frame or can find respondents randomly, like by going to homeless shelters to interview recently evicted people. Its important to note, though, that if youre not using a list frame, youll need to take care to make sure you arent biasing your results by administering the survey in only one place or at only one time a day, thereby missing lots of potential respondents. Its also important to keep in mind that, as with phone surveys, its easy for interviewers to introduce bias by asking questions in a non-uniform way. Face-to-face surveys are best for when you dont have a list frame, for when your questionnaire is complex or requires a lot of explanation, or for when the survey requires personal, visual, or emotional cues. Their strengths and weaknesses are summarized in the chart below:

Face-to-Face Survey Strengths

Face-to-Face Survey Weaknesses

No list frame is required Provides access to populations without addresses, phones, and/or internet Good control over who answers the questions and in what order

High cost and time requirements

Data collection is slow

Interviewer can introduce bias if not careful

Easy to use complex and open-ended questions Allows for incorporation of visual and emotional cues Can decrease non-response rate by allowing interviewer to explain surveys importance

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

HNMCP RESEARCH METHODS HANDBOOK

19

Internet Surveys
Internet surveys are relative newcomers to survey methodology, but they work similarly to mail surveys. The main difference, of course, is that theyre administered online, not on paper and by mail. They can also be more dynamic than mail surveys, since you can include links and animated visual effects. You should consider using an internet survey if you have limited resources, strong computer skills, and are convinced that your population is likely to respond electronically. Their strengths and weaknesses are summarized below:

Internet Survey Strengths

Internet Survey Weaknesses

Low cost, infrastructure, and resource needs Good accessibility to populations with internet Good control over the order of questions answered, depending on survey design Utilization of links and animations make them more dynamic than mail surveys

High nonresponse rates

Some populations do not have internet access

Little control over who answers questions

Difficult to use complex and open-ended questions

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

20

HNMCP RESEARCH METHODS HANDBOOK

Hybrid/Mixed Mode Surveys


Mixed mode surveys combine survey methods, the most common combination being the drop-off survey. In a drop-off survey, the researcher leaves the survey with the respondent personally, but then the respondent replies by mail. This allows the researcher to convey the importance of the survey in person, but also utilizes the strengths of mail surveys. Another common type of hybrid survey combines mail surveys with follow-up phone surveys of those who didnt respond by mail. The biggest strength of hybrid surveys is their ability to increase response rates, either by increasing the number of contacts between the researcher and respondents or by giving the researcher a chance to explain the importance of the survey. On the other hand, combining survey methods can have complicated repercussions for analyzing and comparing responses.

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

HNMCP RESEARCH METHODS HANDBOOK

21

C. Survey Errors
As you decide what survey method to use and get ready to design your survey, its important to be cognizant of survey errors that are easy to make and have serious repercussions for the accuracy of your survey. The four main survey errors are described below:

Coverage Error
Coverage error happens when not all population members have a chance of being surveyed. For example, lets say you use a phone book to administer a phone survey to Massachusetts housing advocates. Any advocate who doesnt have a phone or whose number is unlisted has no chance of being surveyed this is coverage error. To counteract it, make sure that every population member has an equal (or known) chance of being surveyed.

Sampling Error
Sampling error, discussed in the statistics crash course earlier, happens any time you survey only a sample of your target population. Although it cant be entirely avoided, it can be estimated with reasonable accuracy and can be reduced by increasing the sample size.

Measurement Error
Measurement error, also discussed in the statistics crash course above, relates to the quality of data collected, and occurs whenever a response is inaccurate, vague, or cant be compared to other answers. For example, lets say youve included a survey question asking clergy members if they usually lump it when a dispute arises, but you havent explained what it means to lump it. You can expect to receive answers that are vague, left blank, or impossible to compare uniformly this is measurement error. Minimize measurement error by being extremely cautious about what questions you ask, how you ask them, and whether youve inadvertently introduced bias when asking them.

Nonresponse Error
Nonresponse error happens when a significant number of people surveyed do not respond and when those who dont respond are different from those who do in some significant way. For example, you may administer a mail survey to housing advocates across Massachusetts and then discover that a high proportion of advocates in cities with a lot of subsidized housing didnt respond. This might indicate that they are busier and have less time to respond to surveys, which would have an impact on the accuracy of your results. Minimize nonresponse error by making your survey easy to complete for all members of your target population.

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

22

HNMCP RESEARCH METHODS HANDBOOK

D. Sampling
After youve decided who to survey and how to survey them, youll need to decide whether to do a census or whether to survey a sample of your target population. The main reason to sample is to save yourself time and resources if your population is all residents of Massachusetts, then using a sample can save you a lot of both. But youll need to take into account factors other than population size in deciding whether or not to sample. For example, what survey method will you use? If you have a population of 200, then sending a mail survey to all members may make sense. If youre conducting face-to-face surveys, however, interviewing 200 people may be infeasible. Other major considerations are how precise you need to be and the amount of variation within the population. (See the sampling section in the statistics crash course above for more information on how to sample.) Its important to note that taking a census is not inherently more accurate than using a sample. For example, lets say youre conducting a mail survey of all housing advocates in Massachusetts. If you were to do a census, you would contact each advocate, but you would have less time and resources to send preview and follow-up letters, and to call nonrespondents. This could lead to a bigger nonresponse error than you would have if contacting a more manageable sample and engaging in efforts to raise the response rate. Furthermore, unlike the sampling error introduced by using a sample, you would have no way to estimate the nonresponse error of your census survey. In other words, a well-conducted sample survey can be more reliable than a badly conducted census survey.

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

HNMCP RESEARCH METHODS HANDBOOK

23

E. Question Writing
So, youve decided what information you need, who to get it from, and how youre going to get it. Now its time to write your questions. This process can sound intimidating, especially since there is no one recipe for great question writing. Youll have to be careful to avoid introducing bias into your questions, and to ensure that they are written in a way that yields the information you need and that considers the abilities and characteristics of respondents. In this section well review some considerations youll need to take into account, but we encourage you to consult other resources as you write your questions.

Turning Ideas into Questions


To begin, youll need to go back to your reasons for conducting the survey in the first place and translate those information needs into good questions. This involves setting up questions that provide information in a way that can be measured and compared. For example, lets say youre conducting a survey to find out how to improve the Catholic Churchs system for resolving disputes among clergy members. In part, youll want to find out how clergy members feel about the current system, an attitude that can be influenced by a number of factors like how often disputes are resolved under the current system and how often clergy members feel they have to lump it. To get the information youre looking for, then, youll have to write a number of specific questions that together inform you about what youre researching.

Attitudes & Beliefs vs. Behavior & Attributes


Its helpful to be aware of the type of information youre seeking so that you can make sure your questions ask for it. Most information falls into two categories: the first is behaviors and attributes relatively concrete information like age or employment information and the second is attitudes and beliefs. Keep in mind that attitudes and beliefs are difficult to measure, and that different people usually have different perceptions of the same thing or situation. For example, your professor might rate a local restaurant as good, while you might rate it as excellent. Furthermore, your definition of excellent might be something like your professors definition of good. As a result, attitude and belief questions are more susceptible to measurement error than behavior and attribute questions.

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

24

HNMCP RESEARCH METHODS HANDBOOK

F. Question Structure
There are two basic types of question structure: open-ended and close-ended. When youre writing questions, youll need to consider which structure will better capture the information you need.

Open-ended Questions
Open-ended questions provide no answer choices instead, the respondent fills in the answer. For example, in the clergy study you might ask: What is the most common way that you deal with disputes arising between you and other clergy members? Open-ended questions have several advantages and disadvantages, summarized in the chart below:

Open-ended Question Advantages

Open-ended Question Disadvantages

Helpful when researchers dont have enough knowledge to design answer choices or think they have missed an important answer choice Helpful when researchers seek strong opinions or emotions Good for obtaining explanations for answers to other closeended questions Helpful when respondents are unlikely to know an exact answer and must estimate instead

Demanding for respondents

Can produce many different answers about the same topic Tend to produce answers that are imprecise and difficult to compare

More difficult to code and prepare for analysis

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

HNMCP RESEARCH METHODS HANDBOOK

25

Close-ended Questions
Close-ended questions come in several varieties, but they all include a number of answer choices from which the respondent picks. These answer choices should be specific, cover all possible answers, and be mutually exclusive. For example, you might ask the clergy members:
How many disputes have arisen between you and another clergy member over the last year? 1) 2) 3) 4) 5) 0-2 3-5 6-8 9-10 11 or more

This is an example of a close-ended question with ordered answer choices the answer choices move in an ordered way across some spectrum of possible answers. This format is the least demanding for respondents, and usually provides very specific information. You can also write close-ended questions with unordered answer choices, usually when possible answer choices do not fall on a continuum and when you have enough information to provide a complete list of possible answers. These can be particularly helpful for ranking questions. For example, in your research about evictions, you might ask the local housing advocates:
Please rank the most common reasons for tenant evictions in the City of Boston, starting with 1 for the most common and 5 for the least common. Noise Complaints Non-payment of rent Use of unit for illegal purpose Damage to unit Exceeded occupancy limits

A final format for close-ended questions are partially close-ended questions, which include a series of answer choices and an other option that allows respondents to invent an answer. This format can be helpful for generating new information. When youre using close-ended questions, it is worth noting that some research has suggested that people tend to choose from among the first answer choices in written surveys, and among the last answer choices in oral surveys. This effect, called the category order effect, can introduce measurement error. To reduce any such error, try systematically varying the order of answer choices, and avoid listing too many of them.

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

26

HNMCP RESEARCH METHODS HANDBOOK

Some Final Tips


Before we move on from the complex topic of question writing, wed like to give you a few final pointers about common question writing pitfalls: Avoid vaguely worded questions and responses, which usually produce useless information
Rank the dedication of your staff, on a scale of one to five (1 = very dedicated; 5 = totally undedicated).
Problem Who is the respondent thinking of when she thinks of her staff? What is definition of dedicated does the respondent use to answer this question?

Avoid making questions too difficult to answer, like by requesting too much precision or by making respondents do calculations
Please give the percentage of sick days claimed by your staff that fell on Mondays or Fridays.
Problem To answer this question, your respondent will have to count up all sick days for all of her staff, and subsequently divide that number by the number of total sick days that fell on a Monday or Friday. It is unlikely that your respondent will do this calculation accurately

Be cognizant of questions respondents may be not want to answer, like personal questions
Have you or anyone in your family ever had an abortion?
Problem Who wants to answer this kind of personal question?

Avoid using language, such as jargon, abbreviations, and cryptic wording, that respondents are unlikely to understand or can easily misunderstand.
Have you or your business ever resolved a case before the ICC ?
Problem Does ICC refer to the International Criminal Court or the International Chamber of Commerce?

Avoid collapsing questions when they can be easily asked separately


Have you ever felt neglected or unfairly punished by this professor ?
Problem These are actually two separate questions: Have you ever felt neglected by this professor? Have you ever felt unfairly punished by this professor? Asking them in the aggregate gives you no clarity about which of the two questions respondents are answering.

Common sources of bias include:

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

HNMCP RESEARCH METHODS HANDBOOK


o Making a certain behavior or attitude appear normal

27

When is it appropriate to torture suspects?


Problem This question assumes that it is appropriate to torture suspects in some instances. Of course a respondent might answer never, but the mere framing of this question indicates that there must be some instances where it is appropriate.

Using a subjective tone or making objectionable statements in questions

In 2006, only 32% (or less than one third) of respondents indicated that they found sexual relations between two adults of the same sex to be not wrong. The rest of the population found some fault with such relationships, despite the Supreme Courts narrow holding that such relationships are legal. Do you personally object to two consenting adults of the same sex being joined in marriage?
Problem The supposedly factual introduction to this question biases the respondent, by framing opposition to gay marriage as the societal norm, since even mere tolerance of homosexual relationships is seemingly a minority position (according to this biased question). No information is given as to the source of these facts, or how we can be sure that these findings themselves rely on solid research methods.

Answer choices weighted so that there are more choices reflecting one attitude or tendency than another

What factor will weigh most heavily in your choice for president in 2008? a. The poor state of our economy. b. The Republican partys handling of the war in Iraq. c. The Republicans poor record when it comes to environmental protection. d. The lack of quality affordable healthcare in this country. e. This countrys disregard for international diplomacy. f. National security.
Problem While the six options listed all figure into the election, they include many more options pertaining to grievances traditionally held by Democratic voters than grievances held by typically Republican constituencies.

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

28

HNMCP RESEARCH METHODS HANDBOOK

G. Questionnaire Design
Once you have your questions ready, youll need to consider how to present them so that they encourage respondents to complete the survey and are clear and easy to answer. Here we give you some general rules of thumb.

The First Question


Respondents often decide whether or not to complete a survey while reading or listening to the first question, so it should be easy to answer, non-threatening, and clearly related to the stated purpose of the survey. Avoid starting with an open-ended question, and never start with personal questions, like household incomes or demographics; respondents are much more likely to answer this type of question later in the survey.

Question Order
Be careful about how one question can affect the next. Consider, for example, two questions presented consecutively:
1. How satisfied are you with your supervisors dispute management skills? a) b) c) d) e) 2. Very satisfied Somewhat satisfied Neither satisfied nor dissatisfied Somewhat dissatisfied Very dissatisfied

In general, how satisfied are you with your job? a) b) c) d) e) Very satisfied Somewhat satisfied Neither satisfied nor dissatisfied Somewhat dissatisfied Very dissatisfied

The first question gets the respondent thinking about one particular aspect of her job, and then the second question asks her to consider her job overall. If she feels negatively about her boss dispute resolution skills, this can prime her to answer the second question more negatively that she otherwise would have. While there is no one way to deal with this issue, it is helpful to be aware of it in ordering your questions. Testing out your survey design with friends and colleagues, and asking them to comment on possible unwanted biases introduced by your question order, is an important way to minimize such interference.

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

HNMCP RESEARCH METHODS HANDBOOK

29

Question Clusters
Generally speaking, clustering questions about the same topic makes it easier for the respondent to think about that topic. It also helps to cluster questions of the same format together.

Narrative
Using a narrative tone can help respondents feel more comfortable about answering personal questions, and can help signal transitions and changes in topic. For example, it can be helpful to say something like now wed like to ask you a few questions about yourself before asking personal questions.

Oral vs. Written Surveys


Keep in mind that respondents process written questions differently from oral ones. In oral surveys, questions need to be shorter so that they are easier to remember, and you should avoid long lists of answer choices. Written surveys should be easy to read and navigate.

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

30

HNMCP RESEARCH METHODS HANDBOOK

H. Analyzing Results
Once youve administered your survey, its time to analyze your results! This process involves preparing surveys for analysis, analyzing using statistical methods, and then interpreting the results. Here we give you an overview of these three steps.

Cleaning and Coding


The first thing youll need to do is clean and code your surveys to prepare them for analysis. Cleaning a survey simply means looking through it and erasing anything extraneous or clearly erroneous. For example, you might have asked respondents for their year of birth, and one or two may have accidentally written the current year instead. These answers should be discarded, along with any clear inconsistencies in data. Coding responses means assigning numbers to answers choices so they are easy to compare. It is usually easiest to assign codes before administering the survey so that the appropriate numbers can be circled or otherwise marked by either the respondent (in written surveys) or the interviewer (in oral surveys). While close-ended questions are the most straightforward to code, open-ended questions require more work. First youll need to review the answers to develop categories that they all fit into, and then you can go back and code answers according to those categories. You might decide not to code open-ended questions that were mainly ignored, or that served a purely exploratory purpose.

Analysis
Once youve coded your surveys, youll probably want to enter the results into a computer database or software program. Its usually best to arrange answers in a chart so that the rows contain all of the answers of a particular respondent and the columns compare the responses to the same question. To protect confidentiality, avoid naming the rows according to respondents names. A lot of computer programs will calculate measures of central tendency and dispersions for you, since these are the most commonly used statistics (theyre discussed above, in the statistics section). Which statistic to use depends on what information youre looking for. For example, lets say youre interested in how most clergy members currently deal with disputes. A percent distribution would probably be most helpful, enabling you to report that, say, 40% of clergy members say they lump it when they have a dispute, while only 15% say they report the dispute to a superior. If youre interested in how many disputes arose among clergy members in the last year, a mean would probably be helpful, enabling you to say, for example on average, clergy members reported being involved in 5 disputes last year. On the other hand, a dispersion would probably be most helpful for comparing how many disputes arose on average in different churches, enabling you to say, for example, Church A reported 15 disputes last year, while Church B reported only 2. There are no strict guidelines on which statistic to use when, so we encourage you to consult other resources for more guidance if you feel that you need it.

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

HNMCP RESEARCH METHODS HANDBOOK Interpretation


Once youve analyzed your survey results, its time to go back to the question that prompted your survey in the first place. While there is no one way to interpret the results, we recommend stepping back to look at the big picture what big or surprising results popped up that tell you something about your research question? As always, we encourage you to consult with your professors and other outside resources when interpreting your survey results.

31

I. Ethics
There is one other, extremely important topic to think about at every step of your survey design and implementation: ethics. First and foremost, you need to make sure to protect the confidentiality of respondents. Remember, most surveys arent anonymous youre usually working with a list of addresses, phone numbers, employees, and so on, all of which come with names. Confidentiality means that although you can report the identities of respondents, you wont. Always make sure that you report your results in the aggregate, without ever mentioning a specific respondent, and destroy anything that can link responses to particular respondents as soon as its practical. Its also important to notify respondents that the surveys are confidential, not anonymous, and explain how you will protect their confidentiality. Another ethical consideration that can arise with surveys relates to participation. With many surveys, youll need to convince respondents to complete the survey and sometimes contact them multiple times. There is a line between convincing and coercing, however, and you must always respect the right of respondents to decline to participate.

J. References
1. Druckman, D. (2005). Doing Research: Methods of Inquiry for Conflict Analysis. Thousand Oaks, CA: Sage. (Chapter 5) 2. Salant, P. and Dillman, D. (1994). How to Conduct Your Own Survey. London: New York: Wiley.

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

HNMCP RESEARCH METHODS HANDBOOK

33

INTERVIEWS
In the course of your clinical project, you will likely conduct several stakeholder interviews. Prior to interviewing individual stakeholders, you will need to draft an interview protocol, a list of questions to be asked during the interview. As you conduct the interview, your primary goals will be to build trust between yourself and the stakeholder, to gather factual information about the conflict, and to learn about the stakeholders interests. Following each interview, you should type up your notes and send them to the stakeholder to ensure their accuracy. Below are some guidelines for constructing your protocol and conducting a stakeholder interview.

A. General Guidelines
Introduction
Begin the interview by introducing yourself, giving the stakeholder a brief overview of HNMCP, and describing the project to which the interview relates. In general, it is good practice to send the stakeholder a letter of introduction, containing this information, prior to the interview. Doing so will not only save time during the interview, it will give you the chance to provide a thoughtful, thorough, and consistent set of background information to each interviewee. If it is infeasible to send a letter of introduction to each stakeholder, you should tailor the introductory portion of the interview on a case-by-case basis. If a stakeholder already has a significant amount of background information, this can be as short as a sentence or two. Other stakeholders may have little, if any, of this background information. In such cases, you may want to take a little bit more time to describe the project, clarify your role in the process, and provide background and context for the interview. You should then explain to the interviewee the extent to which the information he provides will be kept confidential or will be shared with others. You should describe the specific means by which the interviewees information and identity will be restricted or shared. For example, individual interviewees may not be identified in the final report, but their comments may be categorized by stakeholder group and shared anonymously. The level of confidentiality, and the means by which it is preserved, will vary from project to project. What is important is that the interviewee has a clear idea of how the information from his interview will be used and how his identity with respect to that information will or will not be kept anonymous. You should also assure the interviewee that, after the interview is finished, you will type up and send to him any notes you have taken during the interview, so that he can check them for accuracy.

Questions
Some interviewers prefer to script out the entire interview beforehand, while others prefer to use a bare-bones set of topics and potential questions. In general, we have found it most useful to have a short list of topics for discussion, along with some questions and talking points grouped around each topic. If you feel more comfortable having drafted a full script for the interview, then you should feel free to do so. However, you should not expect to stick precisely to such a script. You should seek to strike a balance between ensuring that you hit the important points of the protocol and allowing the conversation to take its own course, which may reveal important and unanticipated information. As

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

34

HNMCP RESEARCH METHODS HANDBOOK

you ask your prepared questions, you should be ready to ask follow-up questions and request clarification as needed. If you find that you can gather all of the information that you need simply by reading questions from a prepared script, it may be more appropriate and efficient to send out surveys rather than conduct interviews. Your first question should always be: Before we begin, do you have any questions for us? In general, questions should be designed to encourage interviewees to talk at some length about whatever they feel is important. Questions should usually be open-ended, requiring the interviewee to provide more than a yes or no answer. However, it may occasionally be useful to use close-ended questions to clarify a point, test your understanding, or probe a topic. Protocols should be designed so that the interviews can be completed in 30 minutes or less.

Closing
At the end of the interview thank the stakeholder for her/his time and her/his valuable contribution to the process. If you believe that the interviewee has additional valuable information that has not come out, ask if the interviewee is willing to continue the interview for a few minutes longer, or at another time in the near future. If you believe that you have covered all of the relevant information, but the interviewee feels that it is important to share additional information or stories with you, provide them with your email address and encourage them to write out their thoughts and send them to you.

B. Recommended Questions
The questions you will want to include in the protocol will depend on the nature of the project, the stakeholder being interviewed, and the information needed to carry out your project. The suggestions below do not represent an exhaustive list of potential questions. They are provided for your reference and as an aid to developing a protocol that is more extensive and more appropriately tailored to the particular interviewee.

General Questions
In any interview, you should ask some or all of the following questions: Before we begin, do you have any questions for us? What other individuals or organizations might have a stake in this situation? Who else would you recommend we talk to (either within your organization or outside of it) who could give us another perspective on this issue? Before we end this interview, is there anything more you would like to add?

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

HNMCP RESEARCH METHODS HANDBOOK Ongoing Conflict

35

When interviewing a stakeholder for a project that is focused on a specific, ongoing conflict, you may want to include some or all of these questions: When did you first become aware of this conflict? What is the history of the conflict? What issues relating to the situation are important to you, and why? What do you think of the media coverage of the situation? -In what ways has it captured the reality of the situation? -What is the media coverage missing? What would be the best possible outcome for you? Why? What would be the worst possible outcome for you? Why? What do you think are some possible causes of the situation? What measures, if any, are currently being taken to address this situation? -Are those measures effective? Why or why not? -Are those measures satisfactory? Why or why not? Would you be willing to engage in some form of dispute resolution effort designed to address this situation? What are the interests and concerns of other parties to this conflict, as you see them?

No Specified Conflict
When interviewing a stakeholder for a project that does not focus on a specific conflict (for instance, a project focused on redesigning an organizations existing conflict management system), you may want to include some or all of the following questions: Which dispute resolution or conflict management programs currently in place in your organization are working well? -What makes that a successful program? How does your organization measure the success of a dispute resolution or conflict management program? -If your organization does not have specific criteria for success, which criteria would you suggest? Why?

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

36

HNMCP RESEARCH METHODS HANDBOOK

Which dispute resolution or conflict management programs currently in place in your organization are working poorly? -What would you do to improve the program?

Have you personally made use of any of your organizations dispute resolution/conflict management programs? What, if any, measures have you taken to measure or evaluate the effectiveness of your organizations dispute resolution/conflict management programs? When was the current dispute resolution or conflict management program adopted? -What were the circumstances? -Was the program adopted in response to any particular conflict, or set of conflicts? If so, what was it?

How long has the current dispute resolution/conflict management program been in place in this organization? -Has the program changed since you have been with the organization? If so, in what ways? -What prompted the change?

C. Positions to Interests
When a stakeholder states a position in response to an interview question, it is critical to probe deeper to get at the interests underlying that position. Moving a stakeholder from positions to interests is one of the most important, and often one of the most challenging, aspects of a stakeholder interview. There are many reasons why an interviewer may fail to get beneath the stakeholders positions and into his interests. Below are a few of the most common challenges we have observed:

Maintaining Empathy/Rapport: An interviewer may find it difficult to engage in empathic listening while simultaneously probing the stakeholders positions. Many interviewers find themselves falling into an empathic listening mode that makes it easy for them to understand and acknowledge a stakeholders positions, but that makes it more difficult for them to question and push back on those positions at the appropriate time. The interviewer may also fear that by pushing or probing, he will viewed as challenging and hostile, rather than empathic and attentive, which in turn may damage his rapport with the stakeholder, resulting in a less honest, open, or friendly dialogue. Avoiding Conflict/Emotional Escalation: An interviewer may fear that, if she pushes on or probes the stakeholders positions, she will provoke a conflict with the stakeholder. If the stakeholder feels that the interviewer is suspicious, patronizing, or antagonistic, the stakeholder may become defensive or hostile. This is an especially common problem when

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

HNMCP RESEARCH METHODS HANDBOOK

37

the stakeholder is (or may easily become) angry or upset. The interviewer may fear that, by pushing or probing, they will escalate the stakeholders emotions, making a calm person upset or making an upset person irate.

Information Gathering: An interviewer may take a journalistic, rather than conflict assessment, approach to the interview, focusing on gathering factual data instead of assessing the stakeholders interests. The interviewer may be so focused on making sure he understands what he is being told, and paraphrasing or acknowledging the interviewees statements, that he doesnt move on to the step of probing the interviewees statements to get at the underlying interests. Stakeholder Conviction: A stakeholder may deliver his answer to a question with a tone of voice and demeanor that convey a level of certainty or finality that dissuades the interviewer from probing any further. When faced with a statement such as There is no way on Gods green Earth that I will give that slimy, two-faced hack the satisfaction of settling this case out of court, the interviewer may consider it a waste of time to continue questioning the stakeholder about the issue of settlement. Self-Confidence: An interviewer may lack the self-confidence necessary to probe and push back on the interviewees answers. The interviewer may fear that, in the process of probing for interests, hell ask a dumb question or make an off-base remark that will make him look foolish, ignorant, inexperienced in the eyes of the stakeholder. Time Constraints: An interviewer may find that the time it takes to elicit a stakeholders interests on only a few topics would prohibit him from covering other key issues that he needs to hit in the interview. With only thirty minutes available, an interviewer with ten issues to get through may feel that he does not have enough time to spend five or ten minutes digging down to the stakeholders interests on each issue.

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

38

HNMCP RESEARCH METHODS HANDBOOK

FOCUS GROUPS
Usually when people think of focus groups, they imagine groups of people gathered by marketing companies to discuss why they choose a particular brand of some product like soap or laundry detergent over another. But they can also be extremely useful outside of the marketing realm for getting opinions from people from whom you want to learn. Lets say, for example, youre helping a military college figure out how to better integrate women into the traditionally all male student body. Assuming you have no military background yourself, you may have no idea what the difficulties of making such an institution co-ed might be. Convening a few different groups of people affiliated with the college perhaps a group of women students or a group of the colleges administrators and listening to them discuss the issue, might be an ideal way for you to get an initial sense of where the problems lie. Thats exactly what focus groups do: they bring together groups of people and use the interaction among group members to produce data and insights about a topic of interest. This section serves as an introduction to the uses, planning, and execution of focus groups, but its not meant to be a comprehensive guide. If youre planning on using a focus group, we strongly encourage you to consult the resources listed at the end of this section, especially with regard to moderating the group.

A. Uses of Focus Groups


Focus groups are basically group interviews, but instead of being structured as intervieweeresearcher interactions, theyre structured as discussions among group members. The researchers role is to moderate the discussion, keeping it focused on the topic of interest and gearing it toward producing the answers he or she is looking for without getting too involved. Because of this format, the data that focus groups produce is primarily qualitative; youll be hearing about opinions and attitudes, not numbers and statistics. So when should you use a focus group? Generally speaking, theyre best for when you need to do some exploratory research because you dont know much about your topic. More specifically, theyre good for: Learning general information on a topic you know little about Getting new ideas Understanding potential or existing problems with a system, product, etc. Understanding attitudes and perceptions of a system, product, etc. Getting knowledge needed for designing surveys or other quantitative research Generating hypotheses that you can test by some other means Interpreting puzzling results of other research

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

HNMCP RESEARCH METHODS HANDBOOK

39

B. Combining Focus Groups with Other Research Methods


As the above list of focus group uses implies, sometimes focus groups alone can provide the information you need, but other times its helpful to combine them with other research methods. The most common of these combinations is of focus groups and surveys. To illustrate this combination, lets say youre helping a humanitarian aid agency develop a field manual for officers abroad who encounter land disputes and need to help mediate them. Youd like to administer a survey to officers in the field to find out more about their experiences with these disputes, but since most surveys ask specific questions and provide specific answer choices, first you need to get a sense of what questions to ask and what common answers might be. A focus group discussion among officers who have encountered this type of dispute abroad but are currently living in the area can help you get the information you need to write good questions. You might learn, for example, that land disputes in some places tend to be resource based, while in other regions they occur for cultural, ethnic, religious, or economic reasons. This would help you write a survey question that reads:
What is the most common reason for land disputes where you are posted? 1) 2) 3) 4) 5) Resources Cultural Differences Ethnic or religious tensions Economics Other

The focus group would also be a good way to test any questions youve already written to ensure that theyll make sense to field officers, or to learn more about puzzling results of a survey youve already administered. Focus groups are also commonly combined with individual interviewing, either to help researchers choose whom to interview and what questions to ask, or to analyze themes that have already come up in individual interviews. For more information on using focus groups to supplement other research methods, we suggest that you consult the resources listed at the end of this section.

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

40

HNMCP RESEARCH METHODS HANDBOOK

C. Strengths and Limitations of Focus Groups


Like all research methods, focus groups have strengths and limitations that you should take into account when deciding whether to use one as part of your research process. These advantages and disadvantages are summarized in the chart below:

Focus Group Advantages

Focus Group Disadvantages

The open discussion format allows respondents to provide information in their own words.

Use of small numbers of people chosen for specific reasons means results cannot be generalized to larger populations. The open discussion format can make responses difficult to compare and analyze.

Group members can react to and build upon responses of other group members. Focus groups can be used with lots of kinds and combinations of people, ranging from children to high-ranking government officials. Data is produced more quickly and at lower cost than individual interviews. The researcher can interact with respondents, allowing for clarification of ideas, observation of visual cues, etc.

People often act or express opinions differently in groups than when theyre alone.

Dominant group members or other group dynamics can bias results.

The moderator can easily introduce bias when facilitating the group.

The research produced is easy to understand and report.

The researcher has less control over data since its generated by the group.

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

HNMCP RESEARCH METHODS HANDBOOK

41

D. Planning for the Focus Group


Before you can conduct your focus group, youll have to make a lot of decisions. As with any research project, the first of these decisions is what information you need. Since focus groups are usually used when you dont know much about a topic, it can be easy to tell yourself that you can just walk into the discussion unprepared and find out what you find out. Wrong. There is a big difference between not knowing a lot about the topic and not knowing what you want to learn. If you dont discern what information you want from the focus group, the discussion is likely to be too unstructured to give you much usable data.

Choosing Participants
Once you know what information youre looking for, you can decide who you want to participate. The basic principle of choosing focus group participants is that they must be representative of your population of interest, and they must be willing and able to provide the information for which youre looking. They also need to be interested in or have a stake in your topic - otherwise they wont have an engaging discussion. As with surveys, youll generally find participants from some kind of list frame, like an organizational roster. In the military college example, you could choose participants from a class roster, or from the universitys staff directory. Unlike other quantitative methods, however, you wont be generalizing your results to a larger population, so you dont need to worry much about having a random sample. Your list frame just needs to contain representatives of the population youre interested in, and your focus group just needs to contain a few of those representatives.

Number of Groups
How many focus groups you run depends largely on the nature of your research. Generally, the more complex your research question is, or the more diverse your population is, the more focus groups youll need to conduct. To decide how many groups you need, it can be helpful to think about your population as segmented into meaningful subgroups. For example, in the military college project, you might want to conceptualize the population as divided into female students, male students, and administrators, and run three different focus groups one for each subgroup. In the example of field officers with experience handling land disputes, on the other hand, there might not be an obvious or particularly helpful way of subdividing the officers, so you might decide to hold only one focus group. Naturally, how many groups you hold will also depend on your resources. You might ideally want to run five or six groups, but simply not have the time to conduct all of them in one semester. Its worth mentioning that some researchers believe that one focus group per population or subgroup is never enough, since you cant be sure whether the results produced by one group were biased by a particular dynamic of that group. Other researchers argue that since you wont be generalizing your results, this is not a big concern. You should use your best judgment about your research needs and resources, and also consult with your HNMCP clinical supervisor in making a decision about this issue.

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

42 Heterogeneous vs. Homogeneous Groups

HNMCP RESEARCH METHODS HANDBOOK

Embedded in the discussion of how many focus groups to hold is the question of whether your focus groups should be heterogeneous or homogeneous in terms of members identity and role. The level of heterogeneity will almost certainly have an impact on the discussion, so the answer largely depends on your research goals. For example, participants in homogeneous groups tend to be more comfortable discussing sensitive topics, while heterogeneous groups can provide more diverse viewpoints and experiences. So if youre interested in learning about the experiences that female students at the military college have had with hazing, you might be better off with a focus group of only female students, since including male students or administrators would probably discourage them from discussing this potentially embarrassing topic openly. On the other hand, if youre interested in how college administrators react to reports of hazing, you might want to have female students and administrators in the same group. The key is to balance comfort among group members with interesting conversation.

Group Size
The size of the focus group typically between 6 and 12 participants - also has a significant effect on the discussion, particularly its dynamics and structure. Some general characteristics of small and large groups are summarized below:

Small Groups

Large Groups

Helpful for getting information from everyone, since each member has to participate more

Participants can get away with contributing less

More sensitive to individual characteristics of participants

Discussion is harder to manage, requiring more structure and moderator skill Groups can easily break into smaller side conversations

Can be dull

Can require more resources, since researcher will likely have to hold more groups

It can be hard to find facilities to accommodate bigger groups

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

HNMCP RESEARCH METHODS HANDBOOK

43

E. Preparing the Focus Group


Okay, so you know what research questions you need answered and who to contact, and youve decided how many focus groups to hold, how they should be composed, and how big they should be. Now youre ready for some concrete preparations.

Recruiting Participants
When recruiting your participants, you should contact them in person or by phone, mail, or email. Explain the nature of your research and the importance of their participation, and be prepared to give them the time, location and duration of the focus group. If you are planning to record the focus group, you MUST get the participants written consent, even if you plan to destroy the recordings once your research has concluded. Once they agree to participate, you should send them some kind of confirmation, including any directions, maps, or other materials they may need. Its also a good idea to contact them again the day before the focus group to confirm their attendance. Still, youll probably want to recruit a few extra participants, since some people usually drop out or have an emergency at the last minute.

Choosing the Location


The location of the focus group should be somewhere easy for participants to get to; otherwise, they probably wont come! You should make sure that the location is comfortable. If you plan to record the session (and you have gotten participants consent to do so) make sure your location is conducive to any technological needs you might have.

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

44

HNMCP RESEARCH METHODS HANDBOOK

Interview Guide
A key part of your preparation will be the development of an interview guide, which serves to set the agenda for your focus group discussion and to help the moderator ensure that each important topic is addressed. The questions should be derived from your research questions, but they shouldnt be posed as a rigid verbal survey or questionnaire. Instead, the guide should leave flexibility so that the conversation can flow naturally, while providing enough structure to make sure that the most important topics are discussed. Below are just a few general pointers to keep in mind when designing your interview guide; we encourage you to consult outside resources for more guidance. The moderator should memorize the guide to avoid having to refer to a piece of paper while guiding the discussion. Be aware that emotional or stressful topics generally exhaust participants more quickly, meaning that the interview guide should be shorter. Controversial topics discussed in diverse groups are likely to be debated at length, meaning youll probably cover fewer questions. In these situations, the guide should also be shorter. The level of moderator involvement (discussed below) has an impact on guide length too. The more involved the moderator will be, the more questions the group is likely to get through. Make sure questions are posed in language that is easy to understand, and that never puts participants in embarrassing situations. If youre doing more than one focus group, you dont necessarily need to use the same guide, especially if comparing results isnt particularly important for your research.

F. A Word About Group Dynamics


While an in-depth discussion of group dynamics is beyond the scope of this handbook, we think its a good idea to say a word about them before we talk about conducting focus groups, since theyre a fundamental part of focus group research. The overarching theme that you should be aware of is that participants are much more likely to participate actively in your focus group if they feel comfortable in their surroundings. Making participants feel at ease involves being cognizant of personalities in the group and how they affect each other; fostering group cohesion by helping group members get to know each other early in the discussion; and being aware of how environmental considerations like room decorations (which can be distracting) and room arrangements (circles are conducive to conversation) affect the rapport among group members and their levels of participation. This, of course, is just the tip of the iceberg. If you have time, consider consulting an outside resource before conducting your focus group.

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

HNMCP RESEARCH METHODS HANDBOOK

45

G. Conducting the Focus Group


Okay. Youve gathered your participants in an appropriate place, youve set up your recording materials, and youve got the interview guide ready and memorized. Now comes the most complicated part: moderating the focus group. In many focus groups, the moderator is not the researcher, but is instead a professional hired by the researcher. For the purposes of this handbook, however, well assume that you the researcher will be running your own focus group. Accordingly, in this section well give you some guidance about the role of the moderator and some tips for running focus groups, but since this topic is so highly nuanced, we strongly recommend that you consult other resources for more detailed guidance.

The Moderator
The moderator has two main jobs: making sure that the focus group environment is comfortable and non-threatening for participants, and keeping the discussion focused with all participants actively engaged. These tasks primarily mean that you need to size up the group and adjust your leadership style accordingly, assuming different roles as necessary. For example, if youre discussing the hazing experiences of the female students in the military college, as the moderator you may have to spend a lot of time drawing potentially embarrassing information out of participants as sensitively as possible. On the other hand, if youre dealing with college administrators and find that the president of the college is dominating the conversation, you may have to use a stronger hand to keep him quiet and balance out the discussion. This implies that when preparing for the focus group, youll have to prepare both the content of the discussion and the likely nature of the group dynamics, given who the participants are. At the same time, as moderator youll need to balance sensitivity to participants with objectivity, since you can easily introduce bias if youre not careful. There are three main sources of moderator-introduced bias, which can occur consciously or unconsciously. These are: Personal bias when the moderator tends to welcome and reinforce points of view that are consonant with his own. Pleasing the client when the moderator tends to welcome and reinforce points of view that are consonant with those of her client (like the military college administrators who requested that the research be done). Looking for consistency when the moderator tends to welcome and reinforce points of view that are consistent with other research findings.

Typically, these biases are introduced by nodding or commenting favorably on desirable opinions; by being more patient when people are trying to express desirable opinions; or by questioning respondents who hold desirable opinions more than other respondents.

How involved should the moderator be?


At its least structured, a focus group has hardly any moderator participation the moderator kicks off the discussion by introducing the participants and the topic, but after that only interjects when necessary to keep the discussion focused. At its most structured, on the other hand, the moderator

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

46

HNMCP RESEARCH METHODS HANDBOOK

is intensely involved, keeping the discussion tightly controlled and limiting the interaction among group members. Most focus groups fall somewhere in between these extremes, and no matter what, the moderators role is to facilitate the discussion not to simply observe the group or to act as an interviewer. Deciding how involved the moderator should be depends largely on the purposes of your research. Overall, less moderator involvement is good for when you want to generate new ideas or encourage creativity, and more moderator involvement is good for when you want to make sure that all relevant topics are covered, or when you want to make sure they are covered in depth. The two approaches are contrasted below:

Less moderation

More moderation

Creates a free-flowing atmosphere that is conducive to generating new ideas and creativity

Allows for more control over atmosphere

Allows for more group interaction

Detracts from group spontaneity and synergy Ensures that all relevant topics are covered in the desired depth Allows moderator to cut off dominant participants and draw out shy ones

Risks ignoring topics that may be important to the research Allows researcher to see what participants find interesting or important Provides a way for researchers to observe how well they understand the topic Groups are more likely to produce disorganized data, which is harder to analyze Group cohesion and dynamics can break down more easily

Allows moderator to control conversation flow

More room for moderatorintroduced bias

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

HNMCP RESEARCH METHODS HANDBOOK Getting Started

47

The beginning of a focus group sets its agenda and tone, so its important to create a positive atmosphere at the very beginning. This usually involves introducing yourself and your research, as well as setting some ground rules for the discussion. Common ground rules include: one person speaks at a time, no side conversations, treat other participants with respect, and everyone should participate. After the ground rules are laid out, its usually a good idea to have some kind of icebreaker that allows group members to get to know each other; this usually helps them build good rapport and a sense of group, and can also be useful to you as the moderator, since it can give you information about participants that you might be able to use to draw them out later in the discussion. After the participant introductions, you should introduce the discussion topic very generally, since this keeps the conversation broad enough for new ideas to be mentioned and for you to gauge what topics participants find important enough to raise on their own.

Guiding the Discussion


Once the discussion topic is introduced, youre a full-fledged moderator. Keep in mind that moderating is nuanced, and involves personal style, practice, and skill. In this section well introduce a few key bits of advice for successfully guiding the discussion, with the aim of alerting you to just a few of the issues involved in moderating. We very strongly encourage you to consult other resources for additional advice before moderating a focus group.

Making Sure Everyone Participates


From the beginning, you should make it clear that everyones opinion is important and you should encourage all group members to speak throughout the focus group. Awareness of nonverbal cues can sometimes help you draw participants into the conversation. For example, if someone looks puzzled, ask her why. If someone looks insecure about what he has just said, reassure him that it was a valuable contribution. One technique that can be helpful if youre really having trouble with participation is asking each group member to give his or her opinion in turn. This tactic ensures that everyone says something, but since it disrupts the flow of conversation you should avoid using it too often.

Probing
A lot of your job as moderator will involve probing for additional information from participants, asking them to elaborate on ideas that are interesting, that you dont understand, or about which they appear to have more to say. Typically, moderators probe using many of the active listening activities we practiced during the Negotiation Workshop (remember: Acknowledge, Paraphrase, Inquire). When discussants touch on interesting points you would like to learn more about, ask them questions such as tell me more about that. Another useful probing method is reflecting back what youve understood: so, what I heard you say was; this helps you make sure youve understood the persons statement correctly. Remember to push respondents down the ladder of inference and urge them to discuss their interests motivating their positions. Whichever probing method you use, avoid making the person feel defensive or putting them on the spot, like by asking other participants what they think about the persons statement.

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

48 Managing Time

HNMCP RESEARCH METHODS HANDBOOK

Have you ever attended a meeting that you were told would end at a certain time say 6:00 but that was still going at 6:30? If youre like us, you probably start getting antsy, annoyed, or even hostile if youre really late for something else. So, make sure your group ends on time! This means youll have to keep track of what topics have been discussed so that you can be sure that all of the most important issues are covered. Dont jump around from topic to topic, however. Instead, keep the discussion focused on one topic until its sufficiently exhausted and only cut off discussion if its vital that you move on. Another approach is to plan the focus group to be shorter than youve told participants it would be; that way, if it goes longer than you thought, you wont be in trouble with the group members.

Dealing with Common Problems


Experts: Sometimes some group members intimidate others by trying to appear more knowledgeable than everyone else, either because they are or because they think they are. In these situations you should make it clear that youre equally interested in everyones opinion, but be ready to cut the person off or avoid calling on them if they are dominating the conversation. You might do so by transparently acknowledging the persons contribution and thanking them for their active participation, but mentioning that in the interest of time and in order to give everyone a chance to participate, we have to move on. If you ever need a model to learn from, tune in to any episode of OnPoint on National Public Radio and listen to Tom Ashbrook respectfully yet firmly keep the shows many long-winded callers succinct and on point. Alternatively, if the person is genuinely an expert, you might disarm them by calling on them to share their expertise, thereby giving them the role theyre looking for. Friends: If friends participate in the same group, they often have side conversations, refrain from joining the group, or detract from anonymity. If possible, call for a break and respectfully share your observations with the group, and ask them whether they might be willing to help you think of some creative solutions to address the dynamic. Some good solutions that are likely to come up are to physically separate the friends, or for the group to self-monitor its overt behavior. If the dynamic persists and continues to be a distraction, you may ask one of the individuals to leave, and share with them and the rest of the group your reasons for doing so. Hostile group members: Hostile group members tend to make other participants uncomfortable and to affect the discussion negatively. If you find yourself with a hostile group member, it can be helpful to take a short break from the discussion during which you can address the issue directly with the individual. Ask him for feedback on the focus group, and whether there is something you can do to improve the dynamic. Share your observations with him, and avoid attributing negative intentions to him as an individual. Ask him for help brainstorming ways to improving the interaction, assuming he agrees that there is a problem. If this kind of conversation is unproductive, you might minimize his destabilizing impact by acknowledging his presence, but taking a fairly directive lead in managing the focus group (for example, by acknowledging that he is raising his hand but indicating that youd also like to hear other peoples comments). Ejecting the person from the focus group should be a last resort.

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

HNMCP RESEARCH METHODS HANDBOOK

49

H. Analysis
Now that youve moderated and recorded your focus group, youre ready to look at your results. Usually, the first thing youll need to do is transcribe the focus group discussion so that you can use the transcript to analyze the results. Once you do that, there are two basic ways that you can analyze the data: qualitative analysis and content analysis.

Qualitative Analysis
Qualitative analysis, though less formal than content analysis, is usually sufficient for focus groups whose purpose is primarily exploratory. Using this method, youll review the transcript systematically, looking for sections that are relevant to your research questions and then evaluating them in a qualitative way. To illustrate this method, lets go back to the military college example and say that in holding your focus group discussion with women students, one of your primary topics of interest was what hazing experiences, if any, they had had. Over the course of the discussion, all of the students described being hazed by male classmates in one way or another, making it easy to conclude that there is a major hazing problem at the school. You can make this conclusion more specific by systematically reviewing the transcript and identifying parts of the discussion that touch on these hazing experiences, then comparing and contrasting what these experiences were. This method allows you to present qualitative data, like quotations, when reporting the kinds of hazing experienced by the students. The main drawback to this type of qualitative analysis is its reliance on subjective judgments. That is, analyzing data qualitatively means that you will be subjectively choosing which topics to look at and which quotations or other qualitative data to report. It is good to keep this issue in mind while conducting the analysis and reporting the results.

Content Analysis
Content analysis is a way of turning qualitative data into data that can be counted, so it is somewhat more objective than straightforward qualitative analysis. When doing content analysis, youll review your transcript, coding topics of interest as you go so that you can count the number of times that theyre mentioned and then judge their importance. Lets go back to the military college example one more time. This time, lets say that youre interested in finding out how women students who had been hazed felt about the school administrators handling of hazing complaints. To examine this question using content analysis, you would go through the transcript, noting each time the group discussed the administrations handling of hazing incidents and coding these comments according to which words were used to describe the incidents. This would allow you to report, for example, that of 25 mentions of the administrations handling of hazing complaints, 20 were negative, characterized by words like bad or unresponsive, while 3 were neutral and only 2 were positive. In this way, content analysis would allow you back your conclusion that hazing incidents are being handled poorly with countable data. Clearly, this method is not entirely objective, since it still involves making subjective judgments like how to interpret participants language. Other difficulties arise because this method doesnt always differentiate between brief mentions and lengthy discussions of a topic, nor does it take note of who

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

50

HNMCP RESEARCH METHODS HANDBOOK

said what; in the above example, it is not clear whether the same person mentioned hazing 25 times or whether the comments were evenly distributed throughout the group. Adjusting your reporting and keeping these drawbacks in mind can help you present more solid data after your analysis.

I. Ethics
Like other research methods, ethics are of vital importance to focus group research. The primary ethical consideration involved in focus groups is privacy; like surveys, focus groups are confidential, not anonymous, so the identities of participants are known and must be protected. This is particularly a concern in focus groups because youll be recording the participants discussion. Youll have to limit who has access to your recordings, probably to your fellow teammates and others in the clinic, and youll need to take care to destroy any recordings or other data that can link the identities of participants to those recordings as soon as its practical. Likewise, when reporting your results, never attribute a quotation or other data to a particular group member. You should also be sure to tell participants that theyre being recorded and explain the steps youll take to protect their identities. Another ethical concern arises from the interactive nature of focus groups. Since the participants will be interacting with one another, theyll also be providing data and personal information to one another. This means that you should take steps to protect their identities while the group is going on. For example, if youre using nametags, make sure they only include the participants first names.

J. References
1. Morgan, D. (1988). Focus Groups as Qualitative Research. Newbury Park, CA: Sage. 2. Stewart, D. and Shamdasani, P (1990). Focus Groups: Theory and Practice. Newbury Park, CA: Sage.

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

HNMCP RESEARCH METHODS HANDBOOK

51

BIBLIOGRAPHY
Carnevale, P. and De Dreu, C.K.W. (2006). Methods of Negotiation Research. Leiden, The Netherlands: Koninklijke Brill NV. Druckman, D. (2005). Doing Research: Methods of Inquiry for Conflict Analysis. Thousand Oaks, CA: Sage. Gubrium, J.F. and Holstein, J.A., eds. (2002). Handbook of Interview Research: Context & Method. Thousand Oaks, CA: Sage. Jason, L.A., et al, eds. (2004). Participatory Community Research. Washington, DC: American Psychological Association. Morgan, D. (1988). Focus Groups as Qualitative Research. Newbury Park, CA: Sage. Salant, P. and Dillman, D. (1994). How to Conduct Your Own Survey. London: New York: Wiley. Stewart, D. and Shamdasani, P (1990). Focus Groups: Theory and Practice. Newbury Park, CA: Sage. Wright, S. (1979). Quantitative Methods and Statistics. London: Beverly Hills: Sage.

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

52

HNMCP RESEARCH METHODS HANDBOOK

GLOSSARY
Category Order Effect: respondents tend to choose from the first answer choices in written surveys and the last answer choices in oral surveys Census: research that collects data from an entire population Central Limit Theorem: a statistical theorem that states that if a population has at least 600 members and samples drawn from it have at least 30 members, then sample means will follow a normal distribution, even if the population does not. The sample means will, on average, be distributed around the true population mean. If the population does follow a normal distribution, then sample means will too, regardless of population and sample size Close-ended Question: a survey question that provides answer choices Content Analysis: a method of analyzing qualitative data by looking at the content of research and counting occurrences of data of interest Coverage Error: an error that occurs when a sample is drawn from a list frame that does not include all population members Distribution: measures how members of a population break down into categories; a measure of dispersion List Frame: see sampling frame Mean: the average of a data set; a measure of central tendency Measurement Error: an error that occurs when data collected is inaccurate, imprecise, or cannot be compared in any useful way to other data Median: the middle number of a data set; a measure of central tendency Mode: the most common number in a data set; a measure of central tendency Nonprobability Sample: a sample whose members are selected for specific reasons, not randomly Nonresponse Error: a survey error that occurs when a significant number of population or sample members do not respond to the survey and are different from other population or sample members in a way that is important for the study Normal Distribution: a probability distribution of data in which the mean=median=mode; 68% of data falls within 1 standard deviation from the mean; 95% of data falls within 2 standard deviations from the mean; 99% of data falls within 3 standard deviations from the mean Open-ended Question: a survey question that does not provide answer choices Parameter: a characteristic of a population

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

HNMCP RESEARCH METHODS HANDBOOK


Percent Distribution: measures how a population breaks down into categories in terms of percentages; a measure of dispersion Population: all members of a group targeted for study Probability Sample: a sample in which all population members have a known or equal chance of being selected Random Sample: see Probability Sample

53

Range: the difference between the highest and lowest values in a data set; a measure of dispersion Sample: a subset of a population Sampling Distribution: a probability distribution of sample means; follows a normal distribution even if the population does not, as long as the population has at least 600 members and samples have at least 30 members Sampling Error: a calculable error that occurs when research studies only a sample of a population instead of the entire population Sampling Frame: a list of population members from which to draw a sample Standard Deviation: the average difference between an individual data set member and the data sets mean Statistic: a guess of a parameter based on sample data Survey Cleaning: process of erasing anything extraneous or clearly erroneous from completed surveys Survey Coding: process of assigning numbers to survey responses for comparison and analysis

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

You might also like