Professional Documents
Culture Documents
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
TABLE OF CONTENTS
INTRODUCTION ..........................................................................................................1
A. B. What should you take away from this material? ........................................................................................1 How is this guide organized? ........................................................................................................................2
C.
Sampling.........................................................................................................................................................5
When to Sample...................................................................................................................................................................... 5 How to Sample ....................................................................................................................................................................... 5 Probability Samples ............................................................................................................................................................... 6 Random Numbers Table ......................................................................................................................................................... 6 Systematic Sampling............................................................................................................................................................... 7 Cluster Sampling.................................................................................................................................................................... 7 Stratified Sampling................................................................................................................................................................. 7
D. E.
F. G. H. I. J. K. L. M.
Wait a Moment Why am I Reading About Statistics ??? ...................................................................10 Probability ...................................................................................................................................................11 Normal Distribution ....................................................................................................................................11 Sampling Distributions ...............................................................................................................................12 Confidence Intervals ...................................................................................................................................12 Hypothesis Testing ......................................................................................................................................13 Regression ....................................................................................................................................................13 References ....................................................................................................................................................13
SURVEYS.................................................................................................................. 15
A. B. Getting Started ............................................................................................................................................15 Types of Surveys..........................................................................................................................................15
Mail Surveys......................................................................................................................................................................... 16 Telephone Surveys................................................................................................................................................................ 17 Face-to-Face Surveys........................................................................................................................................................... 18 Internet Surveys.................................................................................................................................................................... 19 Hybrid/Mixed Mode Surveys ................................................................................................................................................ 20
C.
D. E.
Turning Ideas into Questions ............................................................................................................................................... 23 Attitudes & Beliefs vs. Behavior & Attributes ...................................................................................................................... 23
F.
G.
H.
I. J.
INTERVIEWS ............................................................................................................ 33
A. General Guidelines ......................................................................................................................................33
Introduction.......................................................................................................................................................................... 33 Questions.............................................................................................................................................................................. 33 Closing ................................................................................................................................................................................. 34
B.
Recommended Questions............................................................................................................................34
General Questions................................................................................................................................................................ 34 Ongoing Conflict.................................................................................................................................................................. 35 No Specified Conflict............................................................................................................................................................ 35
C.
E.
F. G.
A Word About Group Dynamics ...............................................................................................................44 Conducting the Focus Group .....................................................................................................................45
The Moderator ..................................................................................................................................................................... 45 How involved should the moderator be? .............................................................................................................................. 45 Getting Started ..................................................................................................................................................................... 47 Guiding the Discussion ........................................................................................................................................................ 47 Making Sure Everyone Participates ..................................................................................................................................... 47 Probing ................................................................................................................................................................................ 47 Managing Time .................................................................................................................................................................... 48 Dealing with Common Problems.......................................................................................................................................... 48
H.
Analysis ........................................................................................................................................................49
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
I. J.
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
INTRODUCTION
Welcome to the Negotiation and Mediation Clinic! By now you should know what project youll be working on this semester, and youre probably thinking about how you should get started. But as you crack open this handbook, you might be wondering why we want you to learn about research methods. Well, here are some examples of how research methods can help you execute your project and produce solid results: A military college recently became co-ed, but is having trouble retaining its female students. Youd like to learn about the female experience at the college, but you dont even know what kinds of questions to ask. Using a focus group to bring together female students for a discussion could help you figure out why theyre leaving the school. The Catholic Church has found an increasing number of disputes arising among clergy members, and wants to develop a dispute resolution system to help manage them. Administering a survey to clergy members might help you figure out what the current system looks like, and how people within that system would like to see it changed. A large international NGO wants to streamline its decision-making procedures, which are currently ad hoc and decentralized. Conducting interviews of key players within the NGO might help you understand what the current procedures are and which changes are likely to be feasible.
The purpose of this handbook is to give you a brief overview of major research methods as they apply to projects for this clinic. Its not designed to function as an in-depth explanation of these methods, but as an introduction to get you started on the path to reliable research that you can present to your client. Remember, these are real clients with real problems that need real solutions, so we expect you to be responsible in your research and analysis, which might mean consulting additional resources. So with that said, happy researching!
This is of course leaving aside entirely the question of whether perfect research
design is even necessary given the analytical and think tank nature of most of our clinical projects.
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
After reading this manual, you hopefully should be able to do three things: 1. Strategize about your own research methodology: You and your teammates should be able to strategize about how to maximize the rigor of your projects research, taking into consideration your groups limited capacity, time and resources available to carry out and analyze research. 2. Identify your projects research shortcomings: After reading this guide, you should be able to identify ways in which your projects research methodology deviates from the gold standard to which you would be held if you were a PhD candidate or professional statistician. In other words if a statistician or an econometrician were to carefully scrutinize your projects conclusions and research methods, what would she look for when judging whether or not our conclusions were accurate? 3. Speculate about the impact of your research methodologys shortcomings: Finally, you should be able to speculate about the impact of your justifiable research methodology shortcomings. If you collected data in a certain way and suspect that there might be an unavoidable error of some sort, how might your conclusions be different had you somehow been able to correct for that error? Most final projects will include a methodology section of some sort, in which you should specifically put into writing your thinking about the above three points.
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
A. What is Statistics?
So what is statistics? Well, the basic idea is that its a way to learn about a population thats too big to study in its entirety say, all doctors in the United States or all lawyers in China by looking at a sample of that population. But first things first: before we go on, youll need a few key definitions to get you speaking the language of statistics: Population - a population includes all of the individuals in a group you want to study. Individuals dont have to be people; they could be organizations, laws, or even rocks (if youre a geologist)! Sample - a sample is any subset of a population. Parameter - a parameter is a characteristic of a population. For example, if your population were lawyers in China, one parameter would be the percentage that is environmental lawyers. Statistic - a statistic is a guess of a parameter
So, using these definitions, the general idea of statistics is that when you have a population thats too big to examine as a whole, you cant properly analyze its parameters (or characteristics). To get information about the parameters youre interested in, you can look at the characteristics of a more manageable and representative portion, or sample, of the population. Then you can use statistics to make educated guesses about the populations parameters based on that sample. In other words, statistics are educated guesses, based on samples, about population parameters.
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
B. Error
All research methods have room for error and bias, and well discuss these more in the following sections. When doing statistics, though, there are two main errors to be aware of: measurement error and sampling error.
Measurement Error
Measurement error relates to the quality of the data you collect, and is a concern in any research method. With regard to statistics, the idea is that if you collect data from a sample in a way that produces biased or unclear results, it simply cant represent the population accurately. For example, lets say we asked a sample of Harvard students Do you agree that the dining hall is fantastic? A question like this primes respondents to think about the dining hall more positively than they otherwise would, so it would probably produce more yes answers than wed get if we asked the question more neutrally. This, in turn, would cause us to report, based on our sample, that most Harvard students find the dining hall fantastic which probably isnt true. This means that no matter what research method you use, you need to make sure your data is as reliable and clear as possible; otherwise, you wont be able to use it to accurately judge the larger population.
Sampling Error
When you do statistics, the very fact that youre examining only a sample of a population means that youll have some degree of inaccuracy in your research this is sampling error. For example, lets say youre trying to estimate how many of Chinas lawyers specialize in environmental law by looking at graduates of a few major law schools in China. If these law schools represent the population of Chinese lawyers well, they should give you a good idea of what percentage of the population specializes in environmental law. Even so, there is simply no way that the sample can give you a precise number. So no matter what you do, you can never avoid sampling error entirely. What you can do, however, is minimize sampling error by increasing your sample size. The other thing you can do is report it, which most professional researchers do. Youve probably noticed, for example, that when polls results appear in places like newspapers, theyre usually reported as accurate to plus or minus a few percentage points; those percentage points are the sampling error of the poll. While its fortunate that sampling error can be calculated, we dont expect you to calculate it for this class. We do, however, want you to be aware that its there and that its affecting the accuracy of your results.
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
C. Sampling
As youve probably noticed by now, samples are the basis of statistics. Since the point of a sample (at least for statistics) is to generalize its results to an entire population, taking a good sample is fundamental to producing good statistics. In this section, well discuss how to sample and well describe a few of the most common sampling methods.
When to Sample
In almost all cases, you will have to sample. Only when you are analyzing a target group small enough so that you can accurately contact and question every single member of that population, or where your resources are so large that you can again contact and question every single member of a larger population, can you avoid sampling. In this case, you would be conducting a census. Think of the well known US Census, which is carried out every ten years. Estimates project that the 2010 census, which is intended by the Constitution only to enumerate the American population, will cost the US taxpayer $14.5 billion. Imagine the simple task of counting the number of people in the United States costs over 1600 times as much as the entire cost of constructing HLS Northwest corner! Unless your project is dealing with a small target population, or unless the subject matter of your clinical project is REALLY important, it is highly unlikely that HNMCP could ever muster sufficient resources to allow you to conduct a census.
How to Sample
There are three basic steps to sampling: 1. Identify the target population the first thing you need to do is decide who your target population is. In other words, whom are you studying? Is it the Chinese environmental lawyers? Students at HLS? Its helpful to have as precise a definition as possible. 2. Compile a sample frame a sample frame is a list of population members that youll use to draw your sample, like a phone book or a staff directory. Ideally, your sample frame should be complete, up to date and composed only of population members of interest. It should also list each population member only once. These characteristics will protect your research against errors, like choosing the same population member twice or accidentally making it impossible to select some population members. 3. Select the sample all samples fall into two broad categories: probability samples and nonprobability samples. Probability samples are ones chosen randomly, which allows them to be generalized to larger populations. Non-probability samples are ones whose members are chosen for specific reasons, like convenience or role in an organization. Because theyre chosen subjectively, non-probability sample results cant be generalized to the broader population, so we wont focus on them here. Instead, well review characteristics and methods of probability sampling.
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
6 Probability Samples
A probability sample, sometimes called a random sample, is one where each member of the target population has an equal or known possibility of being selected for it. The randomness of this kind of sample, together with the laws of probability, make it likely that it will represent the population; this is what allows the researcher to generalize its results to the larger population with a calculated estimate of error. Its important to stress, however, that just because a particular sample is chosen randomly doesnt mean that its necessarily representative of the population. In fact, its quite possible for probability sampling to produce strange samples. For example, lets say youre studying Chinese environmental lawyers, and youre interested in where theyre from. Although the population as a whole may come disproportionately from Beijing, its entirely possible, although improbable, that you could draw a random sample that contains only lawyers from Shanghai, Guangzhou, and Hong Kong. The value of random sampling lies in probability at a broader level - its highly improbable that if you took ten or twenty more random samples of the Beijing-dominated population you would still end up with samples dominated by lawyers from other cities. There are several ways to take a probability sample, the simplest being drawing sample members from a hat. Well describe a few other common methods below:
To illustrate this method, lets say you want a sample of 50 out of a population of 5000. The first thing you need to do is number the entire population from 1 to 5000. Then, start at the top of the random numbers table and work your way down the columns systematically, selecting any population member whose number appears until you have chosen 50 sample members. Using the above table, you would look at the first 4 digits of each number, since 5000 has 4 digits, choosing population member 3963, 1459, 3505, 0002, and so on.
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
This method relies on the proportion of sample members to population members, using the proportion as an interval to select sample members. For example, if youre choosing a sample of 50 out of a population of 5000, youll be choosing 1 in 100 population members. To use this method, number all of the population members and then choose a starting point randomly. Then go down your population list choosing every 100th population member until you have 50 of them. Before you use this method, make sure that the population list doesnt break down into intervals that might bias your sample. For example, if youre dealing with days of the week and you choose every 7th population member, youll get the same day every time.
Cluster Sampling
This method breaks the population down into clusters most often geographic ones and then samples each cluster randomly. Its helpful for when a good, comprehensive list frame cant be easily compiled, since it allows researchers to construct lists only for clusters being sampled. To illustrate this method, lets say youre researching evictions by the Boston Housing Authority. You would break the city into clusters, like by neighborhood or voting ward, then randomly choose which clusters to sample, and then take random samples of the clusters chosen. Its important to point out that this method is problematic when each cluster is homogeneous within itself, since each one wont accurately represent the population as a whole.
Stratified Sampling
Researchers using this method divide the population into subgroups (or strata), and then take random samples of each one. This technique is helpful for when youre looking at a population as well as some of its subgroups, like if youre studying Chinese lawyers in general, but also want to learn about Chinese environmental lawyers in particular. When using this method, youll have to be aware of what percentage of the overall population each subgroup represents, since sometimes youll want samples to reflect their true representation in the population, but other times youll be more concerned with having samples of equal size. As a result, this technique can have complicated implications for both generalizing back to the overall population and for calculating sampling error.
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
D. Sample Size
When youre getting ready to sample, one important thing to think about is how big your sample should be. Sample size depends on a few factors: Accuracy generally speaking, increasing sample size reduces sampling error, meaning that bigger samples are usually more accurate. You should consider your need for precision when deciding how big your sample should be. Variation populations that have a lot of variation in the characteristics youre studying usually require bigger samples. The idea here is that you want to be confident that any sample you choose would produce roughly the same results, which is unlikely to happen if youre taking small samples from a heterogeneous population. Subgroups - If youre looking at subgroups within a population, youll have to divide your sample into corresponding subgroups. In effect, this creates smaller samples of each subgroup, which increases the sampling error for each one. So keeping accuracy intact for subgroups means using a larger overall sample. Resources Finally, dont forget that youll probably have limited resources, especially time. Bigger samples require more resources, so youll have to find a balance between precision and efficiency.
This is just a quick summary of some important and complicated considerations. Wed recommend looking at the resources listed at the end of this section for more guidance.
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
Median :
Mode :
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
10
Measures of Dispersion
Dispersion looks at the range of data within a data set, so its most useful when you want to understand the variability in your data. For example, we would probably want to use some kind of dispersion to understand what different types of law Chinese lawyers are specializing in and in what numbers. There are several ways of measuring dispersion; well discuss a few of them below: Range : the range is simply the difference between the highest and lowest numbers in a data set. Since its not a calculated number, and since it depends only on the top and bottom extremes of a data set, its not a particularly strong statistical tool, but it does give you a sense of how much variation there is in your data.
Distributions : distributions are more helpful measures of dispersion, and are relatively simple to calculate. First, divide your data set into appropriate categories, and then count how many members of the data set fit into each one. So, for example, you might divide the Chinese lawyers into categories like environmental lawyers, business lawyers, immigration lawyers, etc. Then youd count up how many lawyers fit into each category to get your distribution. Its often most helpful to turn this type of distribution into a percent distribution, which tells you what percent of the population each category represents and thus provides a more readily comparable measure. Calculate the percentages by dividing the number of category members by the total number of data set members. Standard Deviation : standard deviation is the average difference between an individual data set member and the data sets mean. In other words, it tells you how values in a data set spread around the mean. This is one of the most common measures of dispersions, and is an extremely powerful statistical tool. Calculating standard deviation is fairly tedious, and since we dont expect you to do it for this course we wont go into an explanation here. The resources listed at the end of the chapter have both the formula for calculating standard deviation and more information about its uses.
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
11
G. Probability
When we look at a random sample and then translate its results to a population, were relying on probability, defined as the relative frequency with which something is likely to happen. To illustrate the basis for this idea, think of flipping a coin. Theres an equal probability of getting heads or tails, or a probability of 0.5. But if you flip the coin twice, you wont necessarily get heads once and tails once. Instead, to really see heads and tails an equal number of times, youd probably have to flip the coin hundreds of times. This is the idea behind random sampling: if you took a whole lot of random samples, on average theyd represent the population accurately. This favorable probability allows you to make judgments about the population based on a single sample, even though you can never really know if that particular sample represents the population accurately.
Example: Imagine you were given the (incredibly boring) task of flipping a quarter 100 times, and counting up how many times you got heads and tails respectively. Q: how many times would you expect to get heads? The reason you (hopefully) answered both of the above questions with 50, is that we all know that chances of getting heads in a coin toss should be 50/50. If your clinical supervisor asked you to verify this assumption, and you had to conduct the 100 coin-flip exercise 100 times (using a fair coin) you would ultimately be able to chart your results as follows:
y-axis: number of times a 100-coin-flip exercise resulted in a result of x number of heads tosses. number of times you got heads as a result
H. Normal Distribution
The normal distribution is often called a bell curve its a distribution of data that looks like a bell, where the mean, median, and mode are all equal. The area underneath the curve represents probability.
x-axis: Normal distributions are fundamental to statistics because they relate 50 measures of central tendency and dispersion in a key way: the area underneath the curve that lies between one standard deviation in each direction from the mean encompasses 68% of all values in the data set. In other words, there is a 68% probability that any value in a normal distribution will fall within one standard deviation of the mean. Similarly, the area between two standard deviations represents 95% of all values, and the area between One standard three standard deviations represents 99% of all values. deviation
Statisticians calculate the numerical value of the standard deviation, and use it to indicate their certainty 50 in claiming that the findings of their representative sample is either 68%, 95%, or 99% certain to be within x of the findings had we surveyed the entire population (see confidence intervals below). Calculating the numerical value of the standard deviation requires calculations which can be found in any statistics book, but is most likely beyond the scope of a typical HNMCP clinical project.
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
12
I. Sampling Distributions
Imagine you took a sample from a population, and then calculated its mean. Then imagine you took a new sample, and calculated that ones mean. Then imagine that you repeated this over and over again, until you decided to stop and graph all of the sample means you had calculated. What youd have would be a sampling distribution. Even though youd never do this in practice, sampling distributions are fundamental to statistics because they have a key property, which comes from the Central Limit Theorem: as long as youre working with a population with at least 600 members and samples with at least 30 members, then the sample means will be normally distributed around a mean that, on average, is equal to the population mean. This property of sampling distributions allows statisticians to do a lot of things, including making guesses about populations that arent normally distributed based on the characteristics of normal distributions. Thanks to the tireless work of statisticians who tediously proved the truth of the Central Limit Theorem, we can confidently present the findings we calculate from our sample mean, and estimate how likely it is that the true population figures are within x of our findings.
Example: Political Polling. Gallup is one of the most well-known international polling consulting firms. During the 2008 election for U.S. president, Gallup conducted daily surveys of 1000 or more registered U.S. voters. Using the methods described above, Gallup statisticians have estimated that this sample size will give them 95% accuracy within a +/- 3% confidence interval. If instead of 1000 voters Gallup interviews 2000 registered voters for a particular poll, their 95% accuracy confidence interval shrinks to +/- 2%.
J. Confidence Intervals
Even though sampling distributions let statisticians make strong assumptions about how a sample relates to Excerpt from August 28, 2008 poll tracking popularity of John McCain the population, they still can and Obama among registered US voters: never be sure that those assumptions are correct. Survey Methods Confidence intervals are a way For the Gallup Poll Daily tracking survey, Gallup is interviewing no fewer to manage but not eliminate than 1,000 U.S. adults nationwide each day during 2008. this uncertainty. For example, The general-election results are based on combined data from Aug. 25we cant be absolutely sure that 27, 2008. For results based on this sample of 2,723 registered voters, a sample mean is the same as the maximum margin of sampling error is 2 percentage points the population mean. We do, however, know that in a normal (http://www.gallup.com/poll/109897/Gallup-Daily-Obama-Moves-Aheaddistribution, 95% of all values 48-42.aspx) fall within two standard deviations from the mean, and that sample means are normally distributed around the true population mean. So we can construct an interval that is two standard deviations in either direction from the sample mean, and know that 95% of the time, or 95 times out of 100, the population mean will fall within that interval. Its important to highlight, however, that we still have no way of knowing if this particular samples confidence interval contains the population mean. That is, any particular sample could be one of Two the 5 times out of 100 that the population mean is not standard in the confidence interval. deviations
42% (of U.S. electorate support John McCain) (Aug. 25-27, 2008)
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
13
K. Hypothesis Testing
Hypothesis testing is a way of using a sample distribution to test whether a hypothesis is likely to be correct, based on probability. Typically, researchers first create a null hypothesis to be either accepted or rejected, and then an alternative hypothesis, which states the opposite. For example, a null hypothesis might be that the population mean equals 50. The alternative hypothesis, then, would be that the population mean does not equal 50. Researchers then use their sample data to determine how probable it is that the null hypothesis is true; if it appears highly improbable that it is correct, researchers reject the null hypothesis, and if it appears probable they fail to reject the null hypothesis. Of course, since hypothesis testing is based on probability, and we can never be sure how well the sample reflects the population, it is entirely possible to reject a true null hypothesis (a type I error) or to fail to reject a false null hypothesis (a type II error).
L. Regression
Regression analysis is one of the more complex and useful topics in statistics. Essentially, it is a way to use statistics to explain reasons for variance in data. For example, if youre studying professors salaries and you notice that they vary a lot, you might use regression to understand how variables like gender, area of expertise, and number of articles published affect those salaries. The general idea of regression is that if you plot your data on a graph, usually with the independent variable on the x-axis and the dependent variable on the y-axis on a graph, you can draw a line (usually using a computer) through the data that is as close to as many points as possible. This is the regression line, and it gives you information about correlation, or how the two variables move together. It cant, however, tell you about causation; thats for the researcher to hypothesize.
M. References
1. Druckman, D. (2005). Doing Research: Methods of Inquiry for Conflict Analysis. Thousand Oaks, CA: Sage. 2. Newport, F., Saad, L. and Moore D. (1997) How are Polls Conducted? in Where America Stands 2d ed.. John Wiley & Sons, Inc. (available at Gallup.com website - http://media.gallup.com/PDF/FAQ/HowArePolls.pdf) 3. Salant, P. and Dillman, D. (1994). How to Conduct Your Own Survey. London: New York: Wiley. 4. Wright, S. (1979). Quantitative Methods and Statistics. London: Beverly Hills: Sage.
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
15
SURVEYS
Surveys are everywhere these days! You cant stay in a hotel, take a class, or even use the internet without someone asking you to complete a survey. When internet pop-ups ask you to complete a survey about the website youre using, or when the U.S. Census Bureau sends you a profile to fill out, its because youre part of a population from whom these organizations want information. Their surveys are a systematic way of getting that information. The ubiquity of surveys, however, doesnt mean that theyre easy to administer. In fact, there is room for error at every step of the way, from choosing whom to survey to putting the survey questions in order. This section serves as an introduction to basic methods of designing, administering, and analyzing surveys. It is not intended as a comprehensive guide, however you should absolutely consult your professors throughout the survey process, and you may also need to consult the resources listed at the end of this section for further guidance.
A. Getting Started
The first thing you need to decide before doing a survey is what, specifically, you need to learn. Once you know what information youre looking for, you can begin designing the survey. For example, lets say youre helping a legal aid clinic find out why so many of their clients are being evicted by the local housing authority. One thing youll need to find out are the most common reasons for evictions are most people evicted for non-payment of rent? Noise complaints? Something else? Keep in mind that although you might like to know other information like the education level of most of those evicted; you may not need this information to assist the legal clinic. If youre not careful about limiting the amount of it would be nice to know information you seek, your survey will quickly become unwieldy. Once you have a sense of what you need to know, you can figure out who to ask and how to ask them.
B. Types of Surveys
In deciding what type of survey to use, youll need to consider the information you need, the characteristics of the population youre targeting, and your own resources (especially your time, given the length of academic semesters). For example, if youre pressed for time and want your survey results quickly, a phone survey might be the best option for you. On the other hand, if youve decided to target individuals who have been evicted from their homes, they are unlikely to have telephones and addresses; a face-to-face survey would be a better choice in that case. Below is a brief overview of survey methods, along with their major advantages and disadvantages:
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
16
Mail Surveys
As you might imagine, researchers using this method send the survey to a target population or sample by mail, and then respondents mail the completed survey back to the researcher. Researchers typically get addresses from a list frame like a list of utility company customers or a list of employees from a human resources department, taking care to ensure that the list is as complete and accurate as possible. To improve response rates, researchers should send the survey with a cover letter that explains the purpose and importance of the survey and a pre-addressed and stamped envelope. It can also help to send an introductory letter before the survey and a reminder letter (or phone call) after the survey has been sent. Mail surveys are best for when you have limited resources, when you have an accurate address list, when a rapid response time is not required, and when the respondents have the necessary skills to respond accurately in writing. Their strengths and weaknesses are summarized in the chart below:
Low cost, infrastructure and staff needs Good control over how questions are asked High accessibility to most populations (most people have addresses) Gives respondents a sense of privacy
Long data collection time List frames often contain inaccuracies High non-response rates (think: junk mail!) Little control over who responds and in what order they respond to questions Difficult to use complex and open-ended questions, since researcher is not present to provide explanations
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
17
Telephone Surveys
Like mail surveys, phone surveys are conducted using an existing list frame, like the yellow pages or an organizational staff phone directory. The researcher then calls population or sample members using that list, and either administers the survey then or calls back at a more convenient time. (While various kinds of random dialing techniques also exist, you are not likely to use them for this clinic. If you find that you need to, however, we suggest that you consult an appropriate resource.) Unlike mail surveys, though, the surveyor and the respondents interact personally, meaning that you can introduce bias and inconsistencies if youre not careful to present the questions in the same way to everyone. Phone surveys are best for when you need to get responses quickly, when you have an accurate list frame, and when your target population is likely to have phones. Their strengths and weaknesses are summarized in the chart below:
Quick data collection time Good control over who responds to the survey, question order, etc. Low to moderate cost, depending on long distance calls and rates High accessibility to population (most people have phones)
List frames often contain inaccuracies. Interviewer can introduce bias if not careful
No visual communication
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
18
Face-to-Face Surveys
Face-to-face surveys differ from interviews (discussed in the interview section below) in that surveyors administer the same questionnaire in person to a variety of respondents, as opposed to asking a series of questions tailored to the person being interviewed. The surveyor can either use a list frame or can find respondents randomly, like by going to homeless shelters to interview recently evicted people. Its important to note, though, that if youre not using a list frame, youll need to take care to make sure you arent biasing your results by administering the survey in only one place or at only one time a day, thereby missing lots of potential respondents. Its also important to keep in mind that, as with phone surveys, its easy for interviewers to introduce bias by asking questions in a non-uniform way. Face-to-face surveys are best for when you dont have a list frame, for when your questionnaire is complex or requires a lot of explanation, or for when the survey requires personal, visual, or emotional cues. Their strengths and weaknesses are summarized in the chart below:
No list frame is required Provides access to populations without addresses, phones, and/or internet Good control over who answers the questions and in what order
Easy to use complex and open-ended questions Allows for incorporation of visual and emotional cues Can decrease non-response rate by allowing interviewer to explain surveys importance
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
19
Internet Surveys
Internet surveys are relative newcomers to survey methodology, but they work similarly to mail surveys. The main difference, of course, is that theyre administered online, not on paper and by mail. They can also be more dynamic than mail surveys, since you can include links and animated visual effects. You should consider using an internet survey if you have limited resources, strong computer skills, and are convinced that your population is likely to respond electronically. Their strengths and weaknesses are summarized below:
Low cost, infrastructure, and resource needs Good accessibility to populations with internet Good control over the order of questions answered, depending on survey design Utilization of links and animations make them more dynamic than mail surveys
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
20
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
21
C. Survey Errors
As you decide what survey method to use and get ready to design your survey, its important to be cognizant of survey errors that are easy to make and have serious repercussions for the accuracy of your survey. The four main survey errors are described below:
Coverage Error
Coverage error happens when not all population members have a chance of being surveyed. For example, lets say you use a phone book to administer a phone survey to Massachusetts housing advocates. Any advocate who doesnt have a phone or whose number is unlisted has no chance of being surveyed this is coverage error. To counteract it, make sure that every population member has an equal (or known) chance of being surveyed.
Sampling Error
Sampling error, discussed in the statistics crash course earlier, happens any time you survey only a sample of your target population. Although it cant be entirely avoided, it can be estimated with reasonable accuracy and can be reduced by increasing the sample size.
Measurement Error
Measurement error, also discussed in the statistics crash course above, relates to the quality of data collected, and occurs whenever a response is inaccurate, vague, or cant be compared to other answers. For example, lets say youve included a survey question asking clergy members if they usually lump it when a dispute arises, but you havent explained what it means to lump it. You can expect to receive answers that are vague, left blank, or impossible to compare uniformly this is measurement error. Minimize measurement error by being extremely cautious about what questions you ask, how you ask them, and whether youve inadvertently introduced bias when asking them.
Nonresponse Error
Nonresponse error happens when a significant number of people surveyed do not respond and when those who dont respond are different from those who do in some significant way. For example, you may administer a mail survey to housing advocates across Massachusetts and then discover that a high proportion of advocates in cities with a lot of subsidized housing didnt respond. This might indicate that they are busier and have less time to respond to surveys, which would have an impact on the accuracy of your results. Minimize nonresponse error by making your survey easy to complete for all members of your target population.
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
22
D. Sampling
After youve decided who to survey and how to survey them, youll need to decide whether to do a census or whether to survey a sample of your target population. The main reason to sample is to save yourself time and resources if your population is all residents of Massachusetts, then using a sample can save you a lot of both. But youll need to take into account factors other than population size in deciding whether or not to sample. For example, what survey method will you use? If you have a population of 200, then sending a mail survey to all members may make sense. If youre conducting face-to-face surveys, however, interviewing 200 people may be infeasible. Other major considerations are how precise you need to be and the amount of variation within the population. (See the sampling section in the statistics crash course above for more information on how to sample.) Its important to note that taking a census is not inherently more accurate than using a sample. For example, lets say youre conducting a mail survey of all housing advocates in Massachusetts. If you were to do a census, you would contact each advocate, but you would have less time and resources to send preview and follow-up letters, and to call nonrespondents. This could lead to a bigger nonresponse error than you would have if contacting a more manageable sample and engaging in efforts to raise the response rate. Furthermore, unlike the sampling error introduced by using a sample, you would have no way to estimate the nonresponse error of your census survey. In other words, a well-conducted sample survey can be more reliable than a badly conducted census survey.
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
23
E. Question Writing
So, youve decided what information you need, who to get it from, and how youre going to get it. Now its time to write your questions. This process can sound intimidating, especially since there is no one recipe for great question writing. Youll have to be careful to avoid introducing bias into your questions, and to ensure that they are written in a way that yields the information you need and that considers the abilities and characteristics of respondents. In this section well review some considerations youll need to take into account, but we encourage you to consult other resources as you write your questions.
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
24
F. Question Structure
There are two basic types of question structure: open-ended and close-ended. When youre writing questions, youll need to consider which structure will better capture the information you need.
Open-ended Questions
Open-ended questions provide no answer choices instead, the respondent fills in the answer. For example, in the clergy study you might ask: What is the most common way that you deal with disputes arising between you and other clergy members? Open-ended questions have several advantages and disadvantages, summarized in the chart below:
Helpful when researchers dont have enough knowledge to design answer choices or think they have missed an important answer choice Helpful when researchers seek strong opinions or emotions Good for obtaining explanations for answers to other closeended questions Helpful when respondents are unlikely to know an exact answer and must estimate instead
Can produce many different answers about the same topic Tend to produce answers that are imprecise and difficult to compare
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
25
Close-ended Questions
Close-ended questions come in several varieties, but they all include a number of answer choices from which the respondent picks. These answer choices should be specific, cover all possible answers, and be mutually exclusive. For example, you might ask the clergy members:
How many disputes have arisen between you and another clergy member over the last year? 1) 2) 3) 4) 5) 0-2 3-5 6-8 9-10 11 or more
This is an example of a close-ended question with ordered answer choices the answer choices move in an ordered way across some spectrum of possible answers. This format is the least demanding for respondents, and usually provides very specific information. You can also write close-ended questions with unordered answer choices, usually when possible answer choices do not fall on a continuum and when you have enough information to provide a complete list of possible answers. These can be particularly helpful for ranking questions. For example, in your research about evictions, you might ask the local housing advocates:
Please rank the most common reasons for tenant evictions in the City of Boston, starting with 1 for the most common and 5 for the least common. Noise Complaints Non-payment of rent Use of unit for illegal purpose Damage to unit Exceeded occupancy limits
A final format for close-ended questions are partially close-ended questions, which include a series of answer choices and an other option that allows respondents to invent an answer. This format can be helpful for generating new information. When youre using close-ended questions, it is worth noting that some research has suggested that people tend to choose from among the first answer choices in written surveys, and among the last answer choices in oral surveys. This effect, called the category order effect, can introduce measurement error. To reduce any such error, try systematically varying the order of answer choices, and avoid listing too many of them.
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
26
Avoid making questions too difficult to answer, like by requesting too much precision or by making respondents do calculations
Please give the percentage of sick days claimed by your staff that fell on Mondays or Fridays.
Problem To answer this question, your respondent will have to count up all sick days for all of her staff, and subsequently divide that number by the number of total sick days that fell on a Monday or Friday. It is unlikely that your respondent will do this calculation accurately
Be cognizant of questions respondents may be not want to answer, like personal questions
Have you or anyone in your family ever had an abortion?
Problem Who wants to answer this kind of personal question?
Avoid using language, such as jargon, abbreviations, and cryptic wording, that respondents are unlikely to understand or can easily misunderstand.
Have you or your business ever resolved a case before the ICC ?
Problem Does ICC refer to the International Criminal Court or the International Chamber of Commerce?
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
27
In 2006, only 32% (or less than one third) of respondents indicated that they found sexual relations between two adults of the same sex to be not wrong. The rest of the population found some fault with such relationships, despite the Supreme Courts narrow holding that such relationships are legal. Do you personally object to two consenting adults of the same sex being joined in marriage?
Problem The supposedly factual introduction to this question biases the respondent, by framing opposition to gay marriage as the societal norm, since even mere tolerance of homosexual relationships is seemingly a minority position (according to this biased question). No information is given as to the source of these facts, or how we can be sure that these findings themselves rely on solid research methods.
Answer choices weighted so that there are more choices reflecting one attitude or tendency than another
What factor will weigh most heavily in your choice for president in 2008? a. The poor state of our economy. b. The Republican partys handling of the war in Iraq. c. The Republicans poor record when it comes to environmental protection. d. The lack of quality affordable healthcare in this country. e. This countrys disregard for international diplomacy. f. National security.
Problem While the six options listed all figure into the election, they include many more options pertaining to grievances traditionally held by Democratic voters than grievances held by typically Republican constituencies.
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
28
G. Questionnaire Design
Once you have your questions ready, youll need to consider how to present them so that they encourage respondents to complete the survey and are clear and easy to answer. Here we give you some general rules of thumb.
Question Order
Be careful about how one question can affect the next. Consider, for example, two questions presented consecutively:
1. How satisfied are you with your supervisors dispute management skills? a) b) c) d) e) 2. Very satisfied Somewhat satisfied Neither satisfied nor dissatisfied Somewhat dissatisfied Very dissatisfied
In general, how satisfied are you with your job? a) b) c) d) e) Very satisfied Somewhat satisfied Neither satisfied nor dissatisfied Somewhat dissatisfied Very dissatisfied
The first question gets the respondent thinking about one particular aspect of her job, and then the second question asks her to consider her job overall. If she feels negatively about her boss dispute resolution skills, this can prime her to answer the second question more negatively that she otherwise would have. While there is no one way to deal with this issue, it is helpful to be aware of it in ordering your questions. Testing out your survey design with friends and colleagues, and asking them to comment on possible unwanted biases introduced by your question order, is an important way to minimize such interference.
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
29
Question Clusters
Generally speaking, clustering questions about the same topic makes it easier for the respondent to think about that topic. It also helps to cluster questions of the same format together.
Narrative
Using a narrative tone can help respondents feel more comfortable about answering personal questions, and can help signal transitions and changes in topic. For example, it can be helpful to say something like now wed like to ask you a few questions about yourself before asking personal questions.
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
30
H. Analyzing Results
Once youve administered your survey, its time to analyze your results! This process involves preparing surveys for analysis, analyzing using statistical methods, and then interpreting the results. Here we give you an overview of these three steps.
Analysis
Once youve coded your surveys, youll probably want to enter the results into a computer database or software program. Its usually best to arrange answers in a chart so that the rows contain all of the answers of a particular respondent and the columns compare the responses to the same question. To protect confidentiality, avoid naming the rows according to respondents names. A lot of computer programs will calculate measures of central tendency and dispersions for you, since these are the most commonly used statistics (theyre discussed above, in the statistics section). Which statistic to use depends on what information youre looking for. For example, lets say youre interested in how most clergy members currently deal with disputes. A percent distribution would probably be most helpful, enabling you to report that, say, 40% of clergy members say they lump it when they have a dispute, while only 15% say they report the dispute to a superior. If youre interested in how many disputes arose among clergy members in the last year, a mean would probably be helpful, enabling you to say, for example on average, clergy members reported being involved in 5 disputes last year. On the other hand, a dispersion would probably be most helpful for comparing how many disputes arose on average in different churches, enabling you to say, for example, Church A reported 15 disputes last year, while Church B reported only 2. There are no strict guidelines on which statistic to use when, so we encourage you to consult other resources for more guidance if you feel that you need it.
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
31
I. Ethics
There is one other, extremely important topic to think about at every step of your survey design and implementation: ethics. First and foremost, you need to make sure to protect the confidentiality of respondents. Remember, most surveys arent anonymous youre usually working with a list of addresses, phone numbers, employees, and so on, all of which come with names. Confidentiality means that although you can report the identities of respondents, you wont. Always make sure that you report your results in the aggregate, without ever mentioning a specific respondent, and destroy anything that can link responses to particular respondents as soon as its practical. Its also important to notify respondents that the surveys are confidential, not anonymous, and explain how you will protect their confidentiality. Another ethical consideration that can arise with surveys relates to participation. With many surveys, youll need to convince respondents to complete the survey and sometimes contact them multiple times. There is a line between convincing and coercing, however, and you must always respect the right of respondents to decline to participate.
J. References
1. Druckman, D. (2005). Doing Research: Methods of Inquiry for Conflict Analysis. Thousand Oaks, CA: Sage. (Chapter 5) 2. Salant, P. and Dillman, D. (1994). How to Conduct Your Own Survey. London: New York: Wiley.
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
33
INTERVIEWS
In the course of your clinical project, you will likely conduct several stakeholder interviews. Prior to interviewing individual stakeholders, you will need to draft an interview protocol, a list of questions to be asked during the interview. As you conduct the interview, your primary goals will be to build trust between yourself and the stakeholder, to gather factual information about the conflict, and to learn about the stakeholders interests. Following each interview, you should type up your notes and send them to the stakeholder to ensure their accuracy. Below are some guidelines for constructing your protocol and conducting a stakeholder interview.
A. General Guidelines
Introduction
Begin the interview by introducing yourself, giving the stakeholder a brief overview of HNMCP, and describing the project to which the interview relates. In general, it is good practice to send the stakeholder a letter of introduction, containing this information, prior to the interview. Doing so will not only save time during the interview, it will give you the chance to provide a thoughtful, thorough, and consistent set of background information to each interviewee. If it is infeasible to send a letter of introduction to each stakeholder, you should tailor the introductory portion of the interview on a case-by-case basis. If a stakeholder already has a significant amount of background information, this can be as short as a sentence or two. Other stakeholders may have little, if any, of this background information. In such cases, you may want to take a little bit more time to describe the project, clarify your role in the process, and provide background and context for the interview. You should then explain to the interviewee the extent to which the information he provides will be kept confidential or will be shared with others. You should describe the specific means by which the interviewees information and identity will be restricted or shared. For example, individual interviewees may not be identified in the final report, but their comments may be categorized by stakeholder group and shared anonymously. The level of confidentiality, and the means by which it is preserved, will vary from project to project. What is important is that the interviewee has a clear idea of how the information from his interview will be used and how his identity with respect to that information will or will not be kept anonymous. You should also assure the interviewee that, after the interview is finished, you will type up and send to him any notes you have taken during the interview, so that he can check them for accuracy.
Questions
Some interviewers prefer to script out the entire interview beforehand, while others prefer to use a bare-bones set of topics and potential questions. In general, we have found it most useful to have a short list of topics for discussion, along with some questions and talking points grouped around each topic. If you feel more comfortable having drafted a full script for the interview, then you should feel free to do so. However, you should not expect to stick precisely to such a script. You should seek to strike a balance between ensuring that you hit the important points of the protocol and allowing the conversation to take its own course, which may reveal important and unanticipated information. As
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
34
you ask your prepared questions, you should be ready to ask follow-up questions and request clarification as needed. If you find that you can gather all of the information that you need simply by reading questions from a prepared script, it may be more appropriate and efficient to send out surveys rather than conduct interviews. Your first question should always be: Before we begin, do you have any questions for us? In general, questions should be designed to encourage interviewees to talk at some length about whatever they feel is important. Questions should usually be open-ended, requiring the interviewee to provide more than a yes or no answer. However, it may occasionally be useful to use close-ended questions to clarify a point, test your understanding, or probe a topic. Protocols should be designed so that the interviews can be completed in 30 minutes or less.
Closing
At the end of the interview thank the stakeholder for her/his time and her/his valuable contribution to the process. If you believe that the interviewee has additional valuable information that has not come out, ask if the interviewee is willing to continue the interview for a few minutes longer, or at another time in the near future. If you believe that you have covered all of the relevant information, but the interviewee feels that it is important to share additional information or stories with you, provide them with your email address and encourage them to write out their thoughts and send them to you.
B. Recommended Questions
The questions you will want to include in the protocol will depend on the nature of the project, the stakeholder being interviewed, and the information needed to carry out your project. The suggestions below do not represent an exhaustive list of potential questions. They are provided for your reference and as an aid to developing a protocol that is more extensive and more appropriately tailored to the particular interviewee.
General Questions
In any interview, you should ask some or all of the following questions: Before we begin, do you have any questions for us? What other individuals or organizations might have a stake in this situation? Who else would you recommend we talk to (either within your organization or outside of it) who could give us another perspective on this issue? Before we end this interview, is there anything more you would like to add?
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
35
When interviewing a stakeholder for a project that is focused on a specific, ongoing conflict, you may want to include some or all of these questions: When did you first become aware of this conflict? What is the history of the conflict? What issues relating to the situation are important to you, and why? What do you think of the media coverage of the situation? -In what ways has it captured the reality of the situation? -What is the media coverage missing? What would be the best possible outcome for you? Why? What would be the worst possible outcome for you? Why? What do you think are some possible causes of the situation? What measures, if any, are currently being taken to address this situation? -Are those measures effective? Why or why not? -Are those measures satisfactory? Why or why not? Would you be willing to engage in some form of dispute resolution effort designed to address this situation? What are the interests and concerns of other parties to this conflict, as you see them?
No Specified Conflict
When interviewing a stakeholder for a project that does not focus on a specific conflict (for instance, a project focused on redesigning an organizations existing conflict management system), you may want to include some or all of the following questions: Which dispute resolution or conflict management programs currently in place in your organization are working well? -What makes that a successful program? How does your organization measure the success of a dispute resolution or conflict management program? -If your organization does not have specific criteria for success, which criteria would you suggest? Why?
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
36
Which dispute resolution or conflict management programs currently in place in your organization are working poorly? -What would you do to improve the program?
Have you personally made use of any of your organizations dispute resolution/conflict management programs? What, if any, measures have you taken to measure or evaluate the effectiveness of your organizations dispute resolution/conflict management programs? When was the current dispute resolution or conflict management program adopted? -What were the circumstances? -Was the program adopted in response to any particular conflict, or set of conflicts? If so, what was it?
How long has the current dispute resolution/conflict management program been in place in this organization? -Has the program changed since you have been with the organization? If so, in what ways? -What prompted the change?
C. Positions to Interests
When a stakeholder states a position in response to an interview question, it is critical to probe deeper to get at the interests underlying that position. Moving a stakeholder from positions to interests is one of the most important, and often one of the most challenging, aspects of a stakeholder interview. There are many reasons why an interviewer may fail to get beneath the stakeholders positions and into his interests. Below are a few of the most common challenges we have observed:
Maintaining Empathy/Rapport: An interviewer may find it difficult to engage in empathic listening while simultaneously probing the stakeholders positions. Many interviewers find themselves falling into an empathic listening mode that makes it easy for them to understand and acknowledge a stakeholders positions, but that makes it more difficult for them to question and push back on those positions at the appropriate time. The interviewer may also fear that by pushing or probing, he will viewed as challenging and hostile, rather than empathic and attentive, which in turn may damage his rapport with the stakeholder, resulting in a less honest, open, or friendly dialogue. Avoiding Conflict/Emotional Escalation: An interviewer may fear that, if she pushes on or probes the stakeholders positions, she will provoke a conflict with the stakeholder. If the stakeholder feels that the interviewer is suspicious, patronizing, or antagonistic, the stakeholder may become defensive or hostile. This is an especially common problem when
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
37
the stakeholder is (or may easily become) angry or upset. The interviewer may fear that, by pushing or probing, they will escalate the stakeholders emotions, making a calm person upset or making an upset person irate.
Information Gathering: An interviewer may take a journalistic, rather than conflict assessment, approach to the interview, focusing on gathering factual data instead of assessing the stakeholders interests. The interviewer may be so focused on making sure he understands what he is being told, and paraphrasing or acknowledging the interviewees statements, that he doesnt move on to the step of probing the interviewees statements to get at the underlying interests. Stakeholder Conviction: A stakeholder may deliver his answer to a question with a tone of voice and demeanor that convey a level of certainty or finality that dissuades the interviewer from probing any further. When faced with a statement such as There is no way on Gods green Earth that I will give that slimy, two-faced hack the satisfaction of settling this case out of court, the interviewer may consider it a waste of time to continue questioning the stakeholder about the issue of settlement. Self-Confidence: An interviewer may lack the self-confidence necessary to probe and push back on the interviewees answers. The interviewer may fear that, in the process of probing for interests, hell ask a dumb question or make an off-base remark that will make him look foolish, ignorant, inexperienced in the eyes of the stakeholder. Time Constraints: An interviewer may find that the time it takes to elicit a stakeholders interests on only a few topics would prohibit him from covering other key issues that he needs to hit in the interview. With only thirty minutes available, an interviewer with ten issues to get through may feel that he does not have enough time to spend five or ten minutes digging down to the stakeholders interests on each issue.
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
38
FOCUS GROUPS
Usually when people think of focus groups, they imagine groups of people gathered by marketing companies to discuss why they choose a particular brand of some product like soap or laundry detergent over another. But they can also be extremely useful outside of the marketing realm for getting opinions from people from whom you want to learn. Lets say, for example, youre helping a military college figure out how to better integrate women into the traditionally all male student body. Assuming you have no military background yourself, you may have no idea what the difficulties of making such an institution co-ed might be. Convening a few different groups of people affiliated with the college perhaps a group of women students or a group of the colleges administrators and listening to them discuss the issue, might be an ideal way for you to get an initial sense of where the problems lie. Thats exactly what focus groups do: they bring together groups of people and use the interaction among group members to produce data and insights about a topic of interest. This section serves as an introduction to the uses, planning, and execution of focus groups, but its not meant to be a comprehensive guide. If youre planning on using a focus group, we strongly encourage you to consult the resources listed at the end of this section, especially with regard to moderating the group.
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
39
The focus group would also be a good way to test any questions youve already written to ensure that theyll make sense to field officers, or to learn more about puzzling results of a survey youve already administered. Focus groups are also commonly combined with individual interviewing, either to help researchers choose whom to interview and what questions to ask, or to analyze themes that have already come up in individual interviews. For more information on using focus groups to supplement other research methods, we suggest that you consult the resources listed at the end of this section.
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
40
The open discussion format allows respondents to provide information in their own words.
Use of small numbers of people chosen for specific reasons means results cannot be generalized to larger populations. The open discussion format can make responses difficult to compare and analyze.
Group members can react to and build upon responses of other group members. Focus groups can be used with lots of kinds and combinations of people, ranging from children to high-ranking government officials. Data is produced more quickly and at lower cost than individual interviews. The researcher can interact with respondents, allowing for clarification of ideas, observation of visual cues, etc.
People often act or express opinions differently in groups than when theyre alone.
The moderator can easily introduce bias when facilitating the group.
The researcher has less control over data since its generated by the group.
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
41
Choosing Participants
Once you know what information youre looking for, you can decide who you want to participate. The basic principle of choosing focus group participants is that they must be representative of your population of interest, and they must be willing and able to provide the information for which youre looking. They also need to be interested in or have a stake in your topic - otherwise they wont have an engaging discussion. As with surveys, youll generally find participants from some kind of list frame, like an organizational roster. In the military college example, you could choose participants from a class roster, or from the universitys staff directory. Unlike other quantitative methods, however, you wont be generalizing your results to a larger population, so you dont need to worry much about having a random sample. Your list frame just needs to contain representatives of the population youre interested in, and your focus group just needs to contain a few of those representatives.
Number of Groups
How many focus groups you run depends largely on the nature of your research. Generally, the more complex your research question is, or the more diverse your population is, the more focus groups youll need to conduct. To decide how many groups you need, it can be helpful to think about your population as segmented into meaningful subgroups. For example, in the military college project, you might want to conceptualize the population as divided into female students, male students, and administrators, and run three different focus groups one for each subgroup. In the example of field officers with experience handling land disputes, on the other hand, there might not be an obvious or particularly helpful way of subdividing the officers, so you might decide to hold only one focus group. Naturally, how many groups you hold will also depend on your resources. You might ideally want to run five or six groups, but simply not have the time to conduct all of them in one semester. Its worth mentioning that some researchers believe that one focus group per population or subgroup is never enough, since you cant be sure whether the results produced by one group were biased by a particular dynamic of that group. Other researchers argue that since you wont be generalizing your results, this is not a big concern. You should use your best judgment about your research needs and resources, and also consult with your HNMCP clinical supervisor in making a decision about this issue.
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
Embedded in the discussion of how many focus groups to hold is the question of whether your focus groups should be heterogeneous or homogeneous in terms of members identity and role. The level of heterogeneity will almost certainly have an impact on the discussion, so the answer largely depends on your research goals. For example, participants in homogeneous groups tend to be more comfortable discussing sensitive topics, while heterogeneous groups can provide more diverse viewpoints and experiences. So if youre interested in learning about the experiences that female students at the military college have had with hazing, you might be better off with a focus group of only female students, since including male students or administrators would probably discourage them from discussing this potentially embarrassing topic openly. On the other hand, if youre interested in how college administrators react to reports of hazing, you might want to have female students and administrators in the same group. The key is to balance comfort among group members with interesting conversation.
Group Size
The size of the focus group typically between 6 and 12 participants - also has a significant effect on the discussion, particularly its dynamics and structure. Some general characteristics of small and large groups are summarized below:
Small Groups
Large Groups
Helpful for getting information from everyone, since each member has to participate more
Discussion is harder to manage, requiring more structure and moderator skill Groups can easily break into smaller side conversations
Can be dull
Can require more resources, since researcher will likely have to hold more groups
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
43
Recruiting Participants
When recruiting your participants, you should contact them in person or by phone, mail, or email. Explain the nature of your research and the importance of their participation, and be prepared to give them the time, location and duration of the focus group. If you are planning to record the focus group, you MUST get the participants written consent, even if you plan to destroy the recordings once your research has concluded. Once they agree to participate, you should send them some kind of confirmation, including any directions, maps, or other materials they may need. Its also a good idea to contact them again the day before the focus group to confirm their attendance. Still, youll probably want to recruit a few extra participants, since some people usually drop out or have an emergency at the last minute.
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
44
Interview Guide
A key part of your preparation will be the development of an interview guide, which serves to set the agenda for your focus group discussion and to help the moderator ensure that each important topic is addressed. The questions should be derived from your research questions, but they shouldnt be posed as a rigid verbal survey or questionnaire. Instead, the guide should leave flexibility so that the conversation can flow naturally, while providing enough structure to make sure that the most important topics are discussed. Below are just a few general pointers to keep in mind when designing your interview guide; we encourage you to consult outside resources for more guidance. The moderator should memorize the guide to avoid having to refer to a piece of paper while guiding the discussion. Be aware that emotional or stressful topics generally exhaust participants more quickly, meaning that the interview guide should be shorter. Controversial topics discussed in diverse groups are likely to be debated at length, meaning youll probably cover fewer questions. In these situations, the guide should also be shorter. The level of moderator involvement (discussed below) has an impact on guide length too. The more involved the moderator will be, the more questions the group is likely to get through. Make sure questions are posed in language that is easy to understand, and that never puts participants in embarrassing situations. If youre doing more than one focus group, you dont necessarily need to use the same guide, especially if comparing results isnt particularly important for your research.
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
45
The Moderator
The moderator has two main jobs: making sure that the focus group environment is comfortable and non-threatening for participants, and keeping the discussion focused with all participants actively engaged. These tasks primarily mean that you need to size up the group and adjust your leadership style accordingly, assuming different roles as necessary. For example, if youre discussing the hazing experiences of the female students in the military college, as the moderator you may have to spend a lot of time drawing potentially embarrassing information out of participants as sensitively as possible. On the other hand, if youre dealing with college administrators and find that the president of the college is dominating the conversation, you may have to use a stronger hand to keep him quiet and balance out the discussion. This implies that when preparing for the focus group, youll have to prepare both the content of the discussion and the likely nature of the group dynamics, given who the participants are. At the same time, as moderator youll need to balance sensitivity to participants with objectivity, since you can easily introduce bias if youre not careful. There are three main sources of moderator-introduced bias, which can occur consciously or unconsciously. These are: Personal bias when the moderator tends to welcome and reinforce points of view that are consonant with his own. Pleasing the client when the moderator tends to welcome and reinforce points of view that are consonant with those of her client (like the military college administrators who requested that the research be done). Looking for consistency when the moderator tends to welcome and reinforce points of view that are consistent with other research findings.
Typically, these biases are introduced by nodding or commenting favorably on desirable opinions; by being more patient when people are trying to express desirable opinions; or by questioning respondents who hold desirable opinions more than other respondents.
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
46
is intensely involved, keeping the discussion tightly controlled and limiting the interaction among group members. Most focus groups fall somewhere in between these extremes, and no matter what, the moderators role is to facilitate the discussion not to simply observe the group or to act as an interviewer. Deciding how involved the moderator should be depends largely on the purposes of your research. Overall, less moderator involvement is good for when you want to generate new ideas or encourage creativity, and more moderator involvement is good for when you want to make sure that all relevant topics are covered, or when you want to make sure they are covered in depth. The two approaches are contrasted below:
Less moderation
More moderation
Creates a free-flowing atmosphere that is conducive to generating new ideas and creativity
Detracts from group spontaneity and synergy Ensures that all relevant topics are covered in the desired depth Allows moderator to cut off dominant participants and draw out shy ones
Risks ignoring topics that may be important to the research Allows researcher to see what participants find interesting or important Provides a way for researchers to observe how well they understand the topic Groups are more likely to produce disorganized data, which is harder to analyze Group cohesion and dynamics can break down more easily
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
47
The beginning of a focus group sets its agenda and tone, so its important to create a positive atmosphere at the very beginning. This usually involves introducing yourself and your research, as well as setting some ground rules for the discussion. Common ground rules include: one person speaks at a time, no side conversations, treat other participants with respect, and everyone should participate. After the ground rules are laid out, its usually a good idea to have some kind of icebreaker that allows group members to get to know each other; this usually helps them build good rapport and a sense of group, and can also be useful to you as the moderator, since it can give you information about participants that you might be able to use to draw them out later in the discussion. After the participant introductions, you should introduce the discussion topic very generally, since this keeps the conversation broad enough for new ideas to be mentioned and for you to gauge what topics participants find important enough to raise on their own.
Probing
A lot of your job as moderator will involve probing for additional information from participants, asking them to elaborate on ideas that are interesting, that you dont understand, or about which they appear to have more to say. Typically, moderators probe using many of the active listening activities we practiced during the Negotiation Workshop (remember: Acknowledge, Paraphrase, Inquire). When discussants touch on interesting points you would like to learn more about, ask them questions such as tell me more about that. Another useful probing method is reflecting back what youve understood: so, what I heard you say was; this helps you make sure youve understood the persons statement correctly. Remember to push respondents down the ladder of inference and urge them to discuss their interests motivating their positions. Whichever probing method you use, avoid making the person feel defensive or putting them on the spot, like by asking other participants what they think about the persons statement.
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
48 Managing Time
Have you ever attended a meeting that you were told would end at a certain time say 6:00 but that was still going at 6:30? If youre like us, you probably start getting antsy, annoyed, or even hostile if youre really late for something else. So, make sure your group ends on time! This means youll have to keep track of what topics have been discussed so that you can be sure that all of the most important issues are covered. Dont jump around from topic to topic, however. Instead, keep the discussion focused on one topic until its sufficiently exhausted and only cut off discussion if its vital that you move on. Another approach is to plan the focus group to be shorter than youve told participants it would be; that way, if it goes longer than you thought, you wont be in trouble with the group members.
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
49
H. Analysis
Now that youve moderated and recorded your focus group, youre ready to look at your results. Usually, the first thing youll need to do is transcribe the focus group discussion so that you can use the transcript to analyze the results. Once you do that, there are two basic ways that you can analyze the data: qualitative analysis and content analysis.
Qualitative Analysis
Qualitative analysis, though less formal than content analysis, is usually sufficient for focus groups whose purpose is primarily exploratory. Using this method, youll review the transcript systematically, looking for sections that are relevant to your research questions and then evaluating them in a qualitative way. To illustrate this method, lets go back to the military college example and say that in holding your focus group discussion with women students, one of your primary topics of interest was what hazing experiences, if any, they had had. Over the course of the discussion, all of the students described being hazed by male classmates in one way or another, making it easy to conclude that there is a major hazing problem at the school. You can make this conclusion more specific by systematically reviewing the transcript and identifying parts of the discussion that touch on these hazing experiences, then comparing and contrasting what these experiences were. This method allows you to present qualitative data, like quotations, when reporting the kinds of hazing experienced by the students. The main drawback to this type of qualitative analysis is its reliance on subjective judgments. That is, analyzing data qualitatively means that you will be subjectively choosing which topics to look at and which quotations or other qualitative data to report. It is good to keep this issue in mind while conducting the analysis and reporting the results.
Content Analysis
Content analysis is a way of turning qualitative data into data that can be counted, so it is somewhat more objective than straightforward qualitative analysis. When doing content analysis, youll review your transcript, coding topics of interest as you go so that you can count the number of times that theyre mentioned and then judge their importance. Lets go back to the military college example one more time. This time, lets say that youre interested in finding out how women students who had been hazed felt about the school administrators handling of hazing complaints. To examine this question using content analysis, you would go through the transcript, noting each time the group discussed the administrations handling of hazing incidents and coding these comments according to which words were used to describe the incidents. This would allow you to report, for example, that of 25 mentions of the administrations handling of hazing complaints, 20 were negative, characterized by words like bad or unresponsive, while 3 were neutral and only 2 were positive. In this way, content analysis would allow you back your conclusion that hazing incidents are being handled poorly with countable data. Clearly, this method is not entirely objective, since it still involves making subjective judgments like how to interpret participants language. Other difficulties arise because this method doesnt always differentiate between brief mentions and lengthy discussions of a topic, nor does it take note of who
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
50
said what; in the above example, it is not clear whether the same person mentioned hazing 25 times or whether the comments were evenly distributed throughout the group. Adjusting your reporting and keeping these drawbacks in mind can help you present more solid data after your analysis.
I. Ethics
Like other research methods, ethics are of vital importance to focus group research. The primary ethical consideration involved in focus groups is privacy; like surveys, focus groups are confidential, not anonymous, so the identities of participants are known and must be protected. This is particularly a concern in focus groups because youll be recording the participants discussion. Youll have to limit who has access to your recordings, probably to your fellow teammates and others in the clinic, and youll need to take care to destroy any recordings or other data that can link the identities of participants to those recordings as soon as its practical. Likewise, when reporting your results, never attribute a quotation or other data to a particular group member. You should also be sure to tell participants that theyre being recorded and explain the steps youll take to protect their identities. Another ethical concern arises from the interactive nature of focus groups. Since the participants will be interacting with one another, theyll also be providing data and personal information to one another. This means that you should take steps to protect their identities while the group is going on. For example, if youre using nametags, make sure they only include the participants first names.
J. References
1. Morgan, D. (1988). Focus Groups as Qualitative Research. Newbury Park, CA: Sage. 2. Stewart, D. and Shamdasani, P (1990). Focus Groups: Theory and Practice. Newbury Park, CA: Sage.
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
51
BIBLIOGRAPHY
Carnevale, P. and De Dreu, C.K.W. (2006). Methods of Negotiation Research. Leiden, The Netherlands: Koninklijke Brill NV. Druckman, D. (2005). Doing Research: Methods of Inquiry for Conflict Analysis. Thousand Oaks, CA: Sage. Gubrium, J.F. and Holstein, J.A., eds. (2002). Handbook of Interview Research: Context & Method. Thousand Oaks, CA: Sage. Jason, L.A., et al, eds. (2004). Participatory Community Research. Washington, DC: American Psychological Association. Morgan, D. (1988). Focus Groups as Qualitative Research. Newbury Park, CA: Sage. Salant, P. and Dillman, D. (1994). How to Conduct Your Own Survey. London: New York: Wiley. Stewart, D. and Shamdasani, P (1990). Focus Groups: Theory and Practice. Newbury Park, CA: Sage. Wright, S. (1979). Quantitative Methods and Statistics. London: Beverly Hills: Sage.
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
52
GLOSSARY
Category Order Effect: respondents tend to choose from the first answer choices in written surveys and the last answer choices in oral surveys Census: research that collects data from an entire population Central Limit Theorem: a statistical theorem that states that if a population has at least 600 members and samples drawn from it have at least 30 members, then sample means will follow a normal distribution, even if the population does not. The sample means will, on average, be distributed around the true population mean. If the population does follow a normal distribution, then sample means will too, regardless of population and sample size Close-ended Question: a survey question that provides answer choices Content Analysis: a method of analyzing qualitative data by looking at the content of research and counting occurrences of data of interest Coverage Error: an error that occurs when a sample is drawn from a list frame that does not include all population members Distribution: measures how members of a population break down into categories; a measure of dispersion List Frame: see sampling frame Mean: the average of a data set; a measure of central tendency Measurement Error: an error that occurs when data collected is inaccurate, imprecise, or cannot be compared in any useful way to other data Median: the middle number of a data set; a measure of central tendency Mode: the most common number in a data set; a measure of central tendency Nonprobability Sample: a sample whose members are selected for specific reasons, not randomly Nonresponse Error: a survey error that occurs when a significant number of population or sample members do not respond to the survey and are different from other population or sample members in a way that is important for the study Normal Distribution: a probability distribution of data in which the mean=median=mode; 68% of data falls within 1 standard deviation from the mean; 95% of data falls within 2 standard deviations from the mean; 99% of data falls within 3 standard deviations from the mean Open-ended Question: a survey question that does not provide answer choices Parameter: a characteristic of a population
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.
53
Range: the difference between the highest and lowest values in a data set; a measure of dispersion Sample: a subset of a population Sampling Distribution: a probability distribution of sample means; follows a normal distribution even if the population does not, as long as the population has at least 600 members and samples have at least 30 members Sampling Error: a calculable error that occurs when research studies only a sample of a population instead of the entire population Sampling Frame: a list of population members from which to draw a sample Standard Deviation: the average difference between an individual data set member and the data sets mean Statistic: a guess of a parameter based on sample data Survey Cleaning: process of erasing anything extraneous or clearly erroneous from completed surveys Survey Coding: process of assigning numbers to survey responses for comparison and analysis
Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.