RESEARCH ON SOCIAL WORK PRACTICE

Rubin / STANDARDS FOR RIGOR

Standards for Rigor in Qualitative Inquiry

Allen Rubin
President, Society for Social Work and Research

When the Society for Social Work and Research (SSWR) began to sponsor this journal, its board asked that the journal make more explicit that in addition to rigorous quantitative studies, it encourages the submission of rigorous studies that employ qualitative research methods. The journals editor, Bruce Thyer, has met that request, and the manuscripts he has been receiving seem to indicate that some social work researchers have heard the message. Bruce has appointed a number of qualitative researchers to the journals editorial board, effective January 1, 2000. They include Deborah Padgett, James Drisko, Jane Gilgun, Christine Lowery, and Gardenia Harris. These appointments ought to enhance the fairness of appraisals of qualitative studies submitted, and they reflect Bruces commitment to make the journal more inclusive of diverse research methods. I am delighted with this development. Although most of my published work has emphasized quantitative methods, I love rigorous qualitative studies that provide important findings. To wit, a lengthy chapter on qualitative inquiry in the textbook I coauthored (Rubin & Babbie, 1997) identifies quite a few qualitative studies that I admire. Also, I invited SSWR Board member Deborah Padgett, author of the text Qualitative Methods in Social Work (Padgett, 1998), to chair the Abstracts Review Committee for the next SSWR conference, which will convene January 2001 in Atlanta. But I wonder, based on the unevenness in the quality of manuscripts I have reviewed for quite a few journals in recent years, whether prospective authors have sufficient information on what is meant by rigor in qualitative inquiry. Moreover, I doubt that there is sufficient agreement among social scientists in general about standards for rigor in qualitative inquiry. To be sure, I have found as much unevenness in the quality of quantitative manuscripts I have reviewed as well. Unevenness in the rigor of quantitative manuscripts can be more disheartening than unevenness in qualitative manuscripts because there has long been a fairly clear consensus about standards for quality in quantitative inquiry. I find the unevenness in the quality of manuscripts reporting qualitative studies to be more understandable because the use of qualitative methods is growing in the absence of a consensus on standards.
Research on Social Work Practice, Vol. 10 No. 2, March 2000 173-178 2000 Sage Publications, Inc.

173

174

RESEARCH ON SOCIAL WORK PRACTICE

My hunch is that the unevenness in the quality of manuscripts reporting qualitative studies may stem from two problems. One, which I already mentioned, was suggested to me in an e-mail discussion with Deborah Padgett, who noted a lack of consensus (even among those on the same epistemological wavelength) about what standards for rigor are or should be. The other problem may pertain to epistemological differences. Some qualitatively oriented investigators favor radical versions of relativistic epistemologies and believe that a multiplicity of subjective truths implies the absence of objective truth. They may therefore reject any means, even qualitative ones, to pursue objectivity. They may also reject any standards, even qualitative ones, for evaluating the rigor of qualitative studies. In short, they may think that qualitative inquiry means anything goes. I will not repeat my argument with that line of thinking here; readers can see my January 2000 editorial for that. Suffice it to say that rigorous, at least in the context of this journal and in the view of this writer (and reviewer), means that qualitative studies should conform to standards that have been developed for qualitative inquiry. A chief strength of qualitative inquiry is the depth of understanding that it can provide. Manuscripts reporting qualitative studies should not be rushed or superficial. They should report observations and findings in a thorough manner that enables readers to gain an in-depth understanding of the phenomenon being studied and that provides a compelling case for the authors interpretations. Readers should not be left feeling that the authors are merely saying, Trust me. Instead, reportage should conform to standards that provide qualitative analogues for dealing with concerns about the objectivity, reliability, validity, and representativeness of observations. An excellent introductory summary of those standards can be found in Deborahs text. According to Deborah, the key issues in qualitative research are trustworthiness and credibility. Three important threats to trustworthiness and credibility include the distorting effects of (a) reactivity connected to the impact of the researchers presence in the field, (b) researcher biases in the way one asks questions and conducts as well as filters observations, and (c) respondent biases connected largely to social desirability issues. Deborah proposes six strategies for addressing the above threats and thus for enhancing the rigor of qualitative studies. One involves prolonged engagement. This may involve conducting lengthy interviews rather than relying on brief ones, conducting follow-up interviews (perhaps several per respondent), and spending long periods of time in the field building trusting relationships with respondents. Prolonged engagement is thought to lessen the likelihood of reactivity and respondent bias because the researchers presence after a long period of engagement may seem less obtrusive and because respondents may be less inclined or less able to deceive the researcher during

Rubin / STANDARDS FOR RIGOR

175

a prolonged period with a trusting relationship. Lengthier interviews also may improve the interviewers chances of overcoming potential social desirability biases and getting at the truth. A second strategy involves triangulation, in which one attempts to corroborate observations via multiple observational strategies, multiple sources of observations, multiple modes of analysis, and the use of data analysts from different disciplines and with different theoretical orientations. A third strategy involves peer debriefing and support as a mechanism for guarding against the researchers biases. Member checking is a fourth strategy. This involves asking respondents if they agree with the codes and interpretations of the researchers. A fifth strategy involves negative case analysis, in which the researchers conduct a thorough search for cases that do not fit their interpretations. A sixth strategy involves leaving an audit trail of raw data (field notes, interview transcripts, and so on) that others can use to see if your findings are reproducible and verified. In discussing these strategies with me via e-mail, Deborah notes that they are not presented so much as prescriptive demands but as guidelines. According to Deborah, It would be inappropriate for some studies to use some of the strategies. For example, a researcher studying terminal cancer patients may find member checking to be unethical and unfairly burdensome. Similarly, prolonged engagement may not be possible with some target populations. The idea is to apply as many as possible and as are appropriate. I recommend that you read Deborahs text for elaboration of these strategies and for other useful guidance for conducting and reporting rigorous qualitative studies. When I review manuscripts reporting qualitative studies submitted for publication to this journal as well as several other journals, I look to see if the authors used mechanisms, such as the six strategies proposed by Deborah, that attempt to alleviate threats to the trustworthiness and credibility of qualitative inquiry. I also look to see if the way the study was conducted, and the way the findings were reported, fit the in-depth nature of the aims of qualitative inquiry. Alas, usually they do not (just as most of the quantitative reports I review have equally serious flaws that need to be correctedif possible before they merit publication). Some of the qualitative studies that I have reviewed seem to suggest that their authors think that calling their studies qualitative justifies an anything goes mentality. Rather than using prolonged engagement and providing thick descriptions that enable readers to walk in the shoes of their respondents and thus discover new insights and deeper meanings, these studies tend to report disparate brief (usually one line) quotes generated from open-ended questionnaire items or brief interviews with no follow-ups. Rarely in these studies

176

RESEARCH ON SOCIAL WORK PRACTICE

do I see sufficient efforts to triangulate or search for negative cases. Reportage of quantitative aspects of the findings is often vague. Rather than report exactly how many respondents made statements supporting a particular interpretation, the author will say Many respondents said . . . or Often respondents said . . . When I read this sort of reportage, I cannot help but wonder how much is many and how often is often. When authors are vague about this and show no effort to delve thoroughly into cases that do not fit their interpretations, readers might be skeptical about the extent to which the interpretations are influenced by the authors biases (just as they may wonder about uncontrolled biases in quantitative studies). On the other hand, the ambiguity and brevity in some qualitative reports may have less to do with author bias than with a lack of consensus about reportage standards or an insufficient awareness of those standards among authors. Ambiguity also is common in the way sampling and measurement are reported. Granted, using probability sampling procedures to attain representative samples is not expected in qualitative studies. But why not be specific and thorough in reporting the nonprobability sampling procedures that are employed? Some authors just say that theirs was a convenience sample and leave it at that. This leaves important questions unanswered, the qualitative nature of the study notwithstanding. For example, suppose two leading proponents of emphasizing single-case evaluation in research courses conduct and report a qualitative study involving interviews of social work students asking them their attitudes about their research courses. Suppose they just report that it was a convenience sample of 50 students. Their findings indicate that the students found that learning about single-case evaluation was by far the single most beneficial aspect of their entire social work education. Does the fact that the study was qualitative obviate the need for us to know whether, for example, the convenience sample consisted entirely of students currently enrolled in the authors research courses (and hoping for good grades)? I think not. Does the fact that the study was qualitative obviate the need for us to know whether and exactly how many follow-up interviews were conducted? Does it obviate the need to report how the interviews were conducted, by whom, how questions were worded, and so on so as to protect against researcher and respondent biases? Again, I think not. Does the fact that the study was qualitative obviate the need for us to know whether other researchers who do not share the authors enthusiasm for single-case evaluation conducted an independent analysis of the raw data and arrived at the same conclusions. I do not think so. Rigorous qualitative research is labor intensive. It is not a shortcut. It can be harder and more time-consuming than quantitative research. Although qualitative research uses flexible methods, it does not mean anything goes.

Rubin / STANDARDS FOR RIGOR

177

Faculty who are experienced and esteemed qualitative investigators try to convey these caveats to those who take their courses. This may compel some doctoral students who prefer qualitative inquiry to opt for quantitative dissertations so that they can complete their doctoral education more quickly. Other doctoral students, because of the strength of their preferences for qualitative inquiry, might let these caveats go in one ear and out the other. (Those who do so and then attempt to do qualitative dissertations often get a rude awakening! The same goes for students looking for shortcuts around standards for rigor in quantitative dissertations, by the way.) Perhaps the pressure to publish encountered when one joins a faculty is so great that these caveats become forgotten. On the other hand, a more positive possible explanation for the insufficient attention to methodological rigor in some manuscripts may be that adequate strategies for rigor are being used but are just not being reported in the methods sections of manuscripts. In this connection, Deborah mentioned in an e-mail discussion with me, I am continuously dismayed to see what may have been fairly rigorous qualitative studies with methods sections that are completely nonspecific and nondescriptive. Granted, journals have space requirements, but I see little excuse for leaving out a fairly detailed description of the studys methodology and for acknowledging in the strengths and limitations section of the report what strategies for rigor were and were not used (and perhaps why). Whatever the explanation, I urge readers contemplating doing qualitative research to take the above mentioned caveats seriously. If you think that you can get around issues such as bias just by calling your study qualitative, be prepared to have your manuscript rejected (unless, of course, you can submit it to a journal whose editors reject notions of objectivity and standards of rigor). On the other hand, if you have done a rigorous study, be sure to be explicit in reporting its methods, and be sure to thoroughly attend to issues of rigor in both the methods and limitations sections of your manuscript. I hope readers will not misconstrue this editorial as reflecting a dislike for qualitative research. What I dislike is research that lacks rigor. I dislike shoddy studies regardless of the methods they employwhether they are quantitative or qualitative. As I noted at the beginning of this editorial, I love rigorous qualitative studies that provide important findings. So please do not dismiss this editorial as the ravings of an inchworm. Instead, please see it as a constructive effort to increase the submission and eventual publication of rigorous qualitative studies.
Allen Rubin President, Society for Social Work and Research

178

RESEARCH ON SOCIAL WORK PRACTICE

REFERENCES
Padgett, D. K. (1998). Qualitative methods in social work. Thousand Oaks, CA: Sage. Rubin, A., & Babbie, E. (1997). Research methods for social work. Pacific Grove, CA: Brooks/Cole.