How to Prepare a Literature Review

Robert Brown B.Agr.Sc (hons), Ph.D., M.B.A. 10 Kardinia Street, Sunnybank, Brisbane 4109 (Telephone 07 345 4192)

Summary
This is a thumb-nail sketch of some principles that I find helpful in preparing a literature review. Most of it is commonsense (but commonsense is anything but common, I know that because I have learned about the principles from violating them) and comes down to four points. First, do it one paper at a time. Second, keep copies of every document. Third, read what you collect. Fourth, interact with the literature with a pen and highlighter. If you do all that, the review will alsmost write itself.

One Brick at a Time (#1 Golden Rule)

Literature Reviews can seem overwhelming, especially reviews for dissertations. However, as Chairman Mao said, the longest journey starts with a single step, so that golden rule is always to remember that a review is built one document at a time. As you will see in subsequent sections, this means that you need to read about two new papers or book chapters each week and to re-read about the same number of old ones.

The Overall Approach

A literature reveiw is a story about a journey: your journey through scholarship in your particular field. If you remember that it is a story of a journey, a literature review is easier to write. Some of us are seasoned travellers, some of us are not, but even the most seasoned travellers take earch journey only once and, each time, there is something different to be learned from the journey. Moreover, each journey brings forth different things, but never everything. This means that yor review needs to be competent and comprehensive, but not encyclopaedic. Do not feel that you have to be able to write as if you were the great learned guru, because that feeling usually gets in the way. Basic advice about writing a literature review is to set out what is known in a particulr part of a discipline, but that tells only part of the task. The key is to focus on review as well as on literature and this means answering the following questions:

(a) (b) (c) (d) (e) (f) (g)

What is known about the topic? Why is this an important topic? What is unknown? Why are some things unknown? (a good place for speculation) Why should the gaps be filled? Which gaps do you propose to fill and why have you chosen them? How do you propose to fill them?

This is hardly an exhaustive list of the questions that you might pose in a literature reveiw (and the last two would not always be appropriate to a stand-alone review). However, it is important that they be askedd answered explicitly, because they are so often overlooked. In fact, you can do a lot worse that use them as headings in a review because it is so easy to lose sight of how much we know and how little our readers (inclusing examiners) know in comparison to us. When we do this, we also tend to expect that the facts will speak for themselves ( in this case, the facts are the literature cited), but, of couse, they rarely do.

Remember That Literature Is Data

Ever since Pythagoras proposed that everything in the universe could be described in terms of natural numbers, we have had a fascination with numbers that has sometimes caused us to overlook the fact that the numbers are the map, not the territory. Making that error also makes it harder to write a good literature review. In research, there are two types of data: the stuff we generate ourselves and the stuff that others generate, although we often think of them as being more different than they really are. With our own data, we see the raw numbers. With other people's data, we rarelv see anything but processed numbers. Sometimes, it is processed as a set of averages or a correlation, so it is still easily recognisable as data. Sometimes, it is processed into an idea, an insight, or a question. Sometimes, the data is something that cannot even be quantified. For example, "the animals were highly stressed because of the prolonged drought" might be vital data but both the stress and the drought defy strict quantification. The point of this paragraph is to remind you that everythng is data;. The form may vary a little but the substance remains the same. Literature reviews (and other sections of documents) are easier to write if you remember this.

"The Nightly News" versus "Four Corners" or What Makes a Good Review ?
A lot of reviews are just an account of who did what and when and can be as superficial as the six o'clock news. That approach makes a poor review. In contrast, a good review is more like Four Corners. It presents the facts, but it also shows their consequences and implications. A good literature review goes behind the facts to show the issues that have been dealt with in the past, the issues that need to be dealt with today, the gaps and ambiguities in the state of our knowledge, and so on. Sometimes, researchers think that it is not their place to comment and that their role is more like that of some interplanetary observer. Those people usually short-change themselves because the Nightly News approach leaves competence to be inferred from scope of the review whereas the Four Corners approach makes competence more apparent because the analysis and comment proves understanding of the content and issues.

How Many Documents to a Review?

The following are only a rough guide and plenty of good reviews are done outside the suggested ranges. However, be aware that those that fall outside the range may be into overkill or may be rnissing large chunks of relevant literature. Document Expect to cite this many Expect to read this many Doctoral dissertation 100 to 200 700 - 400 Masters dissertation 50 to 100 100 -200 Journal article 12,to25 25-50 Technical note 10 or fewer 10 or fewer Journal review* 50 or more 100 or more (* D. J. de Solla Price's analysis of the scientific literature (Science, vol 149, pp 510 - 515) showed that after every 30 to 40 papers there is need of a review paper to replace those earlier papers that have been loss from sight behind the research front.)

Always Keep Copies of Every Document (#2 Golden Rule)

Keep copies (a) so that you know exact1y what an author said, and (b) so that you can re-read the, document later when you have a better grasp of the topic and will find even more than you did on the first reading. Time spent chasing up a copy that you should have made in the first instance is time wasted.

Always Get the Full Citation Details

Whenever you make a copy, make sure that you get the full citation. Most journals have the journal citation as part of the running heading on alternate pages, but a few have it on only the first page. Either way, make sure it does not get cut off by the photocopier. Articles are useless without citations and remember that not everything can be found on CO ROM if you try to get the citation later on. (For journals, always make sure that you have name and initials for each author, year, title of article, full name of journal, volume number, part number, and page numbers. You will not always use all this inforrnation, but life is a lot easier with it than without it.) With books, always photocopy the tit!e page (and the following page if information about publisher, edition, location, and so on is not all on the title page) and write on it the total number of pages in the book (you may or may not wish to cite the whole book, but it is a lot more efficient to record this information at the outset, especially with inter-library loans, so do it as a matter or routine). If vou copv only parts of chapters, make sure you have the chapter title, author (some books have separate authors for separate chapters) and page numbers of the whole chapter. Apart from photocopying the chapters vou want to cite, it is also a good idea to photocopy the table of contents. As your knowledge of the subject grows, you will often find other chapters that you want to read and and they are easier to retrieve if you make this copy. (If you are worried about copyright, don't be. Copyright laws have always allowed for copying for private study and research, so it is acceptable to copy individual papers and chapters. Copyright law stands to protect the author and publisher from lost income so, if you think you need to copy the whole book, then you should buy the book. On the other hand, if the book is no longer in print, there is no income for the author or publisher to lose, so it becomes harder to argue for a breach of copyright.)

Always Read the Documents You Collect (#3 Golden Rule)

The most expensive shirt you can buy is the one you never wear, even though it may have been a steal at the winter sales. It is the same with literature. You have wasted your time with the paper that you photocopy but never read, yet all of us seem to do this to some extent. As a doctoral student, I often collected papers that looked like they would be useful but put them aside because it seemed to be more important to collect data. Eventually I learned that I would have learned more had I read more of them when I collected them and delayed the data collection a little. Sometimes, it is tempting to think that it is enough to read just the abstracts of the published work, but that is an error. Most abstracts offer little beyond a sumrnary of methods and results, whereas real scholarship is build on an understanding of the issues and options attached to a particular problem. To get these, there is no real alternative to reading the main text. Moreover, an experienced examiner can usuallv guess if a candidate has had a steady diet of abstracts, because the text usuallv lacks depth. (That is not to say that you cannot get away with it, because lots of people do, but they also short-change themselves.)

On the other hand, some documents are not worth reading because they are poorly, written (few journals have the resources to rewrite articles, Nature and Harvard Business Review are a few of the exceptions). The main difficulty you face with these is in deciding whether it is your ignorance of the topic or the author's ignorance of the topic that is causing the problem but, in either case, the remedy is the same. Put the docurnent aside for a few months and try again. Either one of two things will happen: (a) you will find that you have learned enough in the intervening months to come to grips with some difficult material or (b) you will find that you have learned enough to realise that some people do get into print with poorly formed and poorlv expressed ideas and that the document in question is one of them. On the second reading, if you still have difficultv in deading whether a difficult paper is good stuff or rubbish, try to remember that the scholars who know their material best are always the ones who can express it in the simplest way. All great ideas are reallv simple in essence. If a paper is hard to get into, it is often a sign that the authors themselves have not got into the topic as well as they might have and you might be better off not to invest too much time in it.

Read Every Document With a Pen and a Highlighter (#4 Golden Rule)
Usually, this advice is given under the heading of make good notes, but it is more than a matter of making notes. I have yet to meet anyone with a truly photographic memory. Everyone I know seems to have a memorv as poor as my own. Whenever I rummage through my collection of journal articles, I am always surprised that I find papers that I had read with great interest at the time but had since forgotten that I had even seen! Therein lies the argument for making notes, but we need to go beyond it. Notes suggests something sepapate from the document. As a doctoral student, I kept an extensive card index file (I go back to the slide-rule era!). However, even though I wrote lots of notes on some cards, I found that their main use was to help sort docurnents into categories and then to retrieve individual papers (I filed them sequentially and put the number on the card). Whenever, I wanted to write a review I found I always had to go back to the original source because, even with the help of the notes, my memory was never accurate enough. There were always bits that I had overlooked, oversimplified, or distorted. Eventually, I evolved to putting the citation on the card and some overall impressions of the paper just enough to jog my memory. I did the main work on the paper itself: highlighting bits that seemed significant or unusual and writing questions and comments in the margins. Not only did it make it that much easier to find things when I needed them, but it also brought the unexpected benefit of supplying verbatirn quotes that I could use directly in introductions and discussions. In short, my advice to you is to interact with each docurnent you read. Make marks on it left, right and centre. The more you do this, the more it becomes something that you own and can use skilfully rather than just something that you read.

Read All the Important Documents Twice

It is surprising how much extra you will find in a paper when you re-read it six or twelve months later. By that stage, you will almost certainly have a better grasp of the state of play in the your research area and will see questions, issues, and gaps that you had rnissed first time. Again, there is little point in doing this unless you do it with your pen and highlighter in hand. Put things like "rubbish", "important quote", 'I must use this", "Has overlooked Flintstone and Rubble's data", or whatever flags vou need for yourself. Do not be reluctant to check calculations. It is surprising how many errors you can find and how much depth you will add to your review in discussing them.

Stitching Together the Review Step 1: Dump It Under Headings

If you read widely and made good notes on the documents as you read, the review is largely a matter of sewing together a lot of fragrnents. (For the rest of this section, I shall focus on literature reviews for dissertations because they are the biggest ones. The process is the same for other reviews, just on a smaller scale.) The first thing to do is to set out the headings that ought to be covered. Ten is a good number to start with (you probably did this when you started your literature search anyhow), but do not imagine that they are set in cement. As the review evolves, you may well find that some topics merge and new ones appear and that is fine because a review should help you to shape your thoughts, not merely conform to the ones you had when you started. Some people use post-it notes, some write it out long-hand, but my preferred approach is to use the cut-and-paste option in my word-processor. For me, a literature review takes shape out of an initially messy and very long file on my PC. I dump everythng in that I think is relevant, including direct quotes from the text, key bits of the authors' data, an overall synopsis of the document and some of the comments and questions that I have put on the text. At first, the only sign of order is the headings. (I also compile the bibliography at the same time. This achieves two goals. First, it ensures that nothing that I cite in my final draft fails to appear in the list of references. Second, it alerts me to incomplete citations that I may have so that I can avoid the mad scramble at the end to get the missing details and the frustration of having to chop out good references with incomplete citations.) Some articles will span several headings, so use the cut-and-paste option to put them into each heading. Nowhere is it carved in stone that information can be used only once in a review.

(If you are one of those people who managed to get really organised [and perhaps especially if you are one of those people like me who don't], you will find it useful to start this phase at the beginning of the project and add to it as you progress. Apart from forcing you to read more than you might otherwise, it will also give you a good idea as to whether you are getting good coverage of each area and whether there are any important areas that are being overlooked.)

Step 2: The Review-Discussion Two-Step.

If you stop to think about it, a lot of the material covered in a literature review could also be raised in a discussion (and vice versa) so this is a dance in which the literature review leads and the discussion follows. The discussion does different things to the literature review, but the spin you put on the one influences the content of the other. It is as well to be aware of this as you prepare your literature review because it can save you a lot of tirne in preparing your discussion. Accordingly, I always prepare literature reviews or introductions with an eye to the likely shape of the discussion, and, following a waste-not-want-not strategy, I usually find that a lot of w hat I do not use in the review finds a good home in the discussion.

Step 3: String It All Together

Once the inforrnation is sorted into headings, I sort and resort within headings, looking for subheadings, contrasts, contradictions, gaps in the available knowledge, and links between the topics and subtopics. After that, writing the text is largely a matter of putting down sentences to describe the contrasts, contradictions, gaps and links. Especially if I have made good use of verbatim quotes, this section almost writes itself once I reach this stage. (Using verbatim quotes is not as cornmon in the natural sciences as it is in other disciplines and I think that is a pity. I have never yet had a journal editor object to a direct quote. Moreover, I think it is useful to draw a parallel between scientists and lawyers. Much of what we do is to argue cases (most experiments can be seen as cases), and one always argues from a much stronger position when one can quote chapter and verse rather than just a general impression of the content.)

The Embarrassment of Mismatches Between Text and Bibliography

It is unusual if at least one of your examiners or referees does not try to reconcile the references cited in the text with those in the list of references. Discrepancies are both embarrassing and frustrating. If vou are using the Harvard system of citation (author and date), the best way to avoid this problem is to use the search command in your word-processor and search for "19". It will throw up every reference in your text and you can mark off each one on a copy of your bibliography. Any rnismatches will be immediately obvious.

Where Do You Find Relevant Literature?

Gold may be where you find, but a good geologist knows the value of a map, and the best map I know is a CD ROM. The first step is to define your area of interest. Unless you are already expert in the area, this is a lot harder than it sounds and you will probablv need several tries. (To give you an example, I once set out to research the topic "business intelligence and information systerns". I found twelve papers on business intelligence systerns and several thousand papers on business information svstems. Obviously, the first topic was too narrow and the second was too broad.) Most CD ROM databases have a thesaurus function, so use this to get six or so fairly specific topics that define your area of interest. If there are a few gurus who dominate your area, use their names as search terms too. Once you have got that far, searching is largely an iterative matter of searching, reviewing the list that gets retrieved, and adjusting tlte search terms either to narrow or widen the'focus or to shift it to a more fruitful area. Whenever possible, get full abstracts for each item and download each search onto a disc that you can keep. Naturally, keep the search printouts. ROMs are great, but do not overlook hard copies. Computers retrieve only what you tell them to retrieve and there is always useful stuff that you do not know to ask for. Accordingly, make a habit of cruising the racks where new arrivals of each journal are displayed. Take a moment to flick through the contents pages of journals, not just the known heavyweights but also the unfamiliar ones on the margin of your work, because the best work is done by those who cross boundaries. Also, do not overlook abstracting journals and annual reviews. They are slower to use than on-line systems but have the great advantage of laying all their contents before you, not just the bit you can see through an electronic window. When if comes to fine-tuning a search, it is hard to beat the Science Citation Index (and its analogues, the Social Sciences Citation Index and the Arts and Humanities Index). This allows you work backwards, forwards, and sideways across the network of the literature to find every article linked to a given article (and, by extension, articles linked to those articles - the process is limited only by your endurance).

Consolidated or Fragmented?
Some people prefer a separate literature review for each chapter, some prefer to have one large consolidated review, and some do both. Each approach can be made to work well and each has potential weaknesses. If you opt for one large chapter, remember that you will probably have to remind the reader of some issues, especially when the reader is in the later chapters and probablv getting overwhelmed by all the information. Conversely, if you opt for mini-reviews at the start of each chapter, it is hard to avoid writing an overview at the start so that the reader knows the path you intend to take and how the chapters will come together.

Use Lots of Headings and Make Them Specific

This is good advice for any writing, but is doubly applicable to literature reviews because they cover so much territory and, the more the territorv covered, the more the traveller needs roadsigns. Headings are just literary roadsigns and they work better the more common they are and the more specific they are. A series of signs pointing north and saying "Brisbane 1000 krn", "Brisbane 800 km", and so on, works a lot better than an occasional billboard saying "big Queensland city somewhere off that-a-way". It is the same with headings. A general heading saying "Issues associated with this research" is a lot less helpful than a series of headings, one for each issue, with each heading explicitly stating the nature of the issue.

Filing Your Literature

I have to confess that I have yet to find the perfect filing system because so many scholarly articles cross the boundanes of categories. However, here are a few of the options.

Chronological Order.
This is probably the least useful one, unless you have a great memory for what you received and when you received it.

Deep Litter System

This is a less structured version of chronological order. It usually involves dropping reprints into a box, drawer, or folder and rummaging through it from time to time. I use this myself, and I suspect that it is the most common system, but few people care to confess to it.

File by Categories
This is the most logical system, but it falls down because so much literature spans several categories (unless you-set your categories so wide that they are essentially useless) and also because your interests will shift over tirne and some new categories will need to be added, some deleted, some sub-divided, and some amalgamated. My own system spans all three methods. The early literature I collected years ago is still in chronological order in pamphlet boxes backed up with numbered sorter cards sorted into categories. Moreover, I suppose I would still be using this system except that it gets more cumbersome the larger the collection grows. The ability to resort the cards into different categories gets offset bv the greater tirne needed to deal with a larger collection. My later literature starts off in the deep litter systern but moves to categories whenever I find a particularly useful paper. The bottom of the litter pile naturally becomes the papers that I either lack the wit to understand or which are peripheral to my needs. Anyhow, that is what works for me. It may or may not work for you. The thing is to understand that what is really important is to interact with the literature that you accumulate. Unless vou do that, it is no more than scrap paper irrespective of how elegant your filing svstem may be. So try to file it so that you can retrieve, but, above all, read it and re-read it!

10

If You Forget Everything Above ...

Here is the essence of writing a good review: Read - Think - Write - Read - Think Write - Read Think - Write - and keep going until you are done. This is a cyclic process and a good review is rarely written on the first cycle.