SIX RESEARCH FRAMEWORKS

Introduction There are a wide variety of ways in which research can be conducted and the results organized. This paper briefly outlines six of the most common organizational frameworks that are applied to academic research, while acknowledging that there are many other possible frameworks. 1. Problem/Solution Introduction: Problem/solution method is a form of research based on intellectual investigation aimed at discovering, interpreting, and revising human knowledge with the overall aim of resolving a dilemma. Problem/solution research involves identification of a problem or issue that has not been solved. Research into the problem is directed toward finding a solution to the problem/issue and then making recommendations on the feasibility of the identified solution. Problem/solution research is a practical application of research methodology where the goal of the research is primarily a realistic answer to a research question. The problem/solution method has its roots in the policy-analysis process common to the fields of political science and public administration, and is often suitable for military-related topics, too. Simplified, the basic steps are: a) Define the problem b) Set criteria c) Analyze alternatives that meet the set criteria d) Select alternative that best meet the set criteria e) Select that alternative as the solution f) Make recommendations on how to implement selected solution. For a research paper using this approach, following are some steps to help ensure that it is an academically rigorous process: a) Clearly identify the research question b) List research variables and indicators (criteria used to answer the research question) c) Conduct a very thorough literature review to make sure that the question has not already been answered satisfactorily d) Make sure the criteria for a valid, effective solution are clearly justified and established. This can be achieved through a careful description of

measuring instruments, data sources, or strategies for collecting relevant information. e) Make sure that all feasible alternatives have been considered. Pros: A relatively easy research design to understand. Many successful Air Command and Staff College (ACSC) papers have employed this method.

Cons: Maintaining academic rigor can be challenging. If a thorough literature review is not conducted, research papers of this type can often end up repeating previous solutions.

Typical organization: Front matter (title page, disclaimer, table of contents, abstract, etc.) Introduction Description of problem and its key issues Description of what is being measured Discussion of alternatives Results of a comparison between alternative solutions based upon the problems key issues and measurements Recommendations Conclusions Back matter (appendices, endnotes, bibliography, etc.)

2. Quantitative Analysis Introduction: Statistical analysis is primarily a quantitative method because the answers that this type of research produces are primarily numerical values or probabilities. This approach consists of mathematical collection, analysis, interpretation and presentation of data. Statistical analysis takes collected data and summarizes that information in order to make it more usable. Using this method, researchers can make generalizations about a population based on samples drawn from that population. When statistical methods are used to summarize or describe a collection of data this is called descriptive statistics. In contrast, inferential statistics is where patterns in data are modeled in a way that accounts for uncertainty and

randomness in the observations (addresses probabilities), and inferences are drawn from these patterns. Together, descriptive and inferential statistics make up the field known as applied statistics. Applied statistics is used by a wide variety of academic disciplines, from the physical and social sciences to the humanities. Statistical analysis can be simplified into three basic steps: 1) gather the data, 2) organize the data, and 3) analyze the data. One of the most familiar uses of statistics is to determine the chance of some occurrence. This is often referred to as determining a probability. Further explanation of probability and inferential statistics is provided in the Creswell1 or Johnson and Joslyn books.2 Statistical analysis is also used in connection with surveys (see section on surveys, below). Pros: Can produce very accurate and measurable results The same statistical model can often be repeated by other researchers, thus demonstrating one of the key platforms of the scientific method--that experiments can be replicated with identical results given the same input.

Cons: The development of statistical models can be both time- and costprohibitive Often requires a strong mathematical background.

Typical organization: Front matter (title page, disclaimer, table of contents, abstract, etc.) Introduction/the research question Background/literature review Methodology/explanation of the level and type of statistical method used Results of the analysis Discussion of the analysis results Recommendations Conclusions Back matter (appendices, endnotes, bibliography, etc.)

3. Scenario Planning Introduction: Scenario planning is a method used to make predications about the future so that long-term plans can be developed in anticipation of those predications. Scenario

planning examines an issue in greater depth and identifies the driving forces that could shape the research topic. Driving forces are the forces of change outside your organization that will shape future dynamics in both predictable and unpredictable ways, and they can be specific to the internal working environment or broad like social mores, technological trends, economic realities, environmental issues, and political systems or policies.3 The possible outcomes derived from different combinations of these driving forces then become the scenarios upon which long-term plans can be based. Scenario planning is a very useful tool for futures research, allowing researchers to explore potential future challenges and threats. Scenario planning was best defined and explained in Peter Schwartzs book, The Art of the Long View.4 In this method, researchers try to make predictions of what the future might look like by assuming that certain socio-economic driving factors will continue, and then combining those driving factors with one or two uncertain forces. From the interaction of these projections scenariosalso called alternative futuresone can better understand what the future may hold. Schwartz identified eight basic steps to conducting solid scenario planning (paraphrased, simplified and quoted from the Appendix of The Art of the Long View): a) b) c) d) Identify the focal issue List key factors influencing the focal issue List driving forces from the larger world that influence these key factors Rank these key factors and driving forces on the basis of two criteria: 1. The degree of importance 2. The degree of uncertainty Select the most likely factors and scenarios with greatest impact Flesh-out these selected scenarios Discuss the implications of these scenarios Try to select the scenarios that are most likely and important for recommendations.5

e) f) g) h)

Schwartz also identifies five general rules of thumb to help ensure welldeveloped scenario plans (also paraphrased, simplified and quoted from the same Appendix in The Art of the Long View): a) Try to avoid too few or too many scenarios (four scenarios seem to generally work well) b) In general, try to avoid assigning probabilities to different scenarios; sometimes the most improbable scenario may be the most important to consider

c) Pay a great deal of attention to naming the scenarios (if scenario names are provocative or memorable, the scenarios will have a much better chance of making their way into the decision-making process) d) When developing scenarios, be imaginative but ensure that imagination is tempered by reality; do not develop scenarios that cannot possibly occur. e) The best scenarios tend to be those that are both plausible and surprising.6 Since much of the scenario planning-based research method looks at future possibilities and issues, the resulting research can be thought of as case studies for the future. Still, some argue that the basic principles of scenario planning can be used to develop new outcomes for past historical events. This branch of history, called counter-factual history, delves into the exciting world of what might have been. For example, what would have been the outcome of World War II if Hitler had taken the advice of his generals and not attacked the Soveit Union? Another excellent source of information on scenario planning is the Scearce and Fulton reading, What If? The Art of Scenario Thinking for Non-profits. Scearce and Fultons scenario-thinking model consists of five phases: Orient, Explore, Synthesize, Act, and Monitor. Their model provides a framework for researchers to look beyond near-term concerns and urgent requirements, and to gaze far ahead and create plausible alternative futures that will stimulate discussion and preparation.7 Pros: Cons: Like any type of method that tries to anticipate future issues and concerns, it is imperfect Survey may be inappropriate when just looking at one, specific, singledimension question (the Delphi method works better for this) Individuals and organizations can get a false sense of security in regard to future issues and concerns after this method has been successfully employed a couple of times. In short, they begin to believe that a possible future is the actual future. This, of course, could lead to catastrophe. Can be a very useful method when trying to figure out the most likely future scenarios that an organization might face One of the most reliable methods for trying to predict future issues of concern Very good at determining complex forecasts involving multiple factors.

Typical organization: Front matter (title page, disclaimer, table of contents, abstract, etc.) Introduction/the research question Background/literature review Methodology Results Discussion of the results Recommendations Conclusions Back matter (appendices, endnotes, bibliography, etc.)

4. Case Study Introduction: A case study is one way of doing social science research. Rather than using large samples and following a rigid protocol to examine a limited number of variables, case study research involves an in-depth examination of a single instance or event--commonly called a case. Case studies provide a systematic way of looking at events, collecting data, analyzing information, and reporting the results. As a result, the researcher may gain a sharpened understanding of why an incident happened as it did, and what might become important to look at more extensively in future research.8 There are three common types of case study research designs exploratory, descriptive, and explanatory. a) Exploratory case study research models are used when the goal is to develop a hypotheses for further investigation (attempts to answer what or so what? questions) b) Descriptive case study research models try to describe key players and issues in a particular case (attempts to answer the who and where? questions) c) Explanatory research models try to identify causation (attempts to answer the how and why? questions). Case study research has been a common research strategy in medicine, business, psychology, sociology, public administration, political science, planning and economics. It can also be used to assess a particular military operation or issue. In all of these fields, the need for case studies has come from a desire to understand a complex episode. A key advantage of this method is that it allows the investigation to retain meaningful characteristics of actual events.

Case studies can use a traditional research paper format or they can use more of a story-telling type of format (the outline given below is for a traditional case study research format). The story-telling format is often found in historical case studies. These seldom have the distinct sections described below; however, they typically have the same general flow as the traditional case study and research paper design. Pros: Cons: Academic rigor can be easily lost because of equivocal evidence or the researchers biased views It is difficult to create valid scientific generalizations based upon a single case (thus, the recent growth of the comparative case study method which examines several cases) Case studies have a tendency to produce large amounts of data and can quickly become unreadable. Can provide detailed description of key events Can provide unique insight into dynamic social interactions Can provide a wealth of detail.

Traditional organization: Front matter (title page, disclaimer, table of contents, abstract, etc.) Introduction Literature review In-depth analysis of the case study Discussion of the case study results Recommendations Conclusions Back matter (appendices, endnotes, bibliography, etc.)

5. Surveys NOTE: All surveys conducted by ACSC students must adhere to AFI 362601, Air Force Personnel Survey Program and ACSC OI 36-19, Survey Control Program. The survey approval process normally takes 2-3 months to complete. No survey data collection can begin before actual approval is received. A thorough review and understanding of these instructions are required before selecting a survey as a research framework. Introduction:

Survey research is commonly used in the sciences, marketing, government, health, education and political fields. Survey research is used to collect information about variables within a population (for this reason, surveys often contain sensitive personal information which has to be carefully safeguarded). It can focus on numerical data, yes/no answers or opinions. Survey research usually consists of four main parts: 1) the development of the survey questions, 2) the selection of the survey group, 3) the administration of the survey, and 4) analysis of the results.

There are six typical ways of conducting a survey: E-mail/postal questionnaires Internet questionnaire (web-based, on-line questionnaire) On-the-spot questionnaire Face-to-face interviews Focus groups Telephone interviews.

There are advantages and disadvantages to each of these survey types. This discussion will focus on the pros and cons of e-mail/postal questionnaires, faceto-face interviews and focus groups since those are most common types of surveys that have been employed by Air Force researchers. E-mail/postal questionnaires Pros: This method is very good for issues that are difficult to talk about in person, it is good for collecting quantitative data, and this type of survey can reach a large geographical area and a large number of people relatively quickly. Also, e-mail surveys are particularly cost effective. Not only are the mailing costs eliminated, but processing costs are reduced, too. E-mail questionnaires can be designed in such a way that the responses are automatically loaded into the appropriate software. Additionally, people are used to completing these types of surveys, and the questionnaires can be completed by the survey subjects at a time and place of their convenience. Cons: There is always a need to motivate the target population to complete and return the survey instrument, but participation can be enhanced by building instruments that allow subjects to complete their surveys at a time and place of their choosing. Receiving clarification on specific questions within the instrument is typically unavailable during completion. These surveys also require an up-to-date address list. Finally, postal questionnaires can be expensive and responders must be able to read and write in order to participate.

Face-to-face interviews Pros: With this method surveyors are able to get more qualitative data, they can explore answers with the respondents, and they can read nonverbal feedback from their subjects. In addition, it is easier to ask longer and more detailed questions. Typically these type of interviews result in a higher rate of survey completion and, because surveyors can assist respondents with unfamiliar words, questions or concepts (interviewees do not need to be able to read, see and write to participate), the results can be more inclusive. Cons: Interviews are time consuming for the both the interviewer and subject and some individuals may feel uncomfortable sharing personal information in person. Also, interviewers must find suitable places to conduct their interviews. Last, there is a personality dimension to the results that needs to be taken into consideration. There is, for instance, the potential for interviewer bias (both verbal or non-verbal), as well as the human tendency of subjects to try to provide answers they feel will satisfy their interviewer. Focus group Pros: This type of survey is not as time consuming as individual, face-toface interviews and works better for defining group norms and themes. The interviewer can: read non-verbal group feedback; expect high rates of survey completion; explore answers with respondents; and assist respondents with unfamiliar words, questions or concepts (interviewees do not need to be able to read, see and write to participate). Cons: This type of survey creates an even more uncomfortable setting for sharing personal information than face-to-face interviews. Small-group bias may taint results, whether expressed verbally or non-verbally. Interviewers must find suitable places to conduct the group interview, and there remains the possibility of interviewer bias. After collecting survey results researchers usually summarize their data using descriptive statistics. Descriptive statistics is a basic way of summarizing data generated by a survey. This summary usually takes the form of percentages of responses for each possible answer to a question. Typical organization: Front matter (title page, disclaimer, table of contents, abstract, etc.) Introduction/the research question Background/literature review Methodology/explanation of survey method used

Results of survey Analysis of results Recommendations Conclusions Back matter (appendices, endnotes, bibliography, etc.)

6. Evaluation The process of evaluation can be defined as assigning a value to, appraising, or carefully examining something. In research, evaluation involves determining the worth, importance, or impact of something and often implies an ethical connotation. If the evaluation is based on ethics or morality then an assessment of the good/bad or morality/immorality of something is ascertained. Consequently, evaluations should be based on some sort of standardized moral criteria. Standardized criteria are also required for evaluations that are not based on ethics. For example, if researchers were evaluating the utility of different bomber aircraft they might create standard criteria for evaluation as follows: a) b) c) d) e) Range Payload Maintenance requirements Speed Stealthiness.

Evaluations usually result in some form of conclusions or recommendations. The conclusions or recommendations should be based on how well the evaluation subjects meet the grading criteria. There are many quantitative methods for grading criteria to include prioritized standards, equal points for each grade, or unbalanced scoring based on a second set of criteria. Regardless of the criteria used, the process must be clear, easy to explain, and pertinent to the subject being evaluated. Similar to the problem/solution approach, some steps to help ensure that evaluations are academically rigorous are to: a) Clearly identify the research topic b) List and explain the criteria that will be used to evaluate the research topic c) Conduct a very thorough literature review to make sure someone else has not evaluated this topic recently d) Make sure the criteria for evaluation are clearly justified and established:

1. Carefully describe measuring instruments, data sources, or strategies for collecting relevant information 2. Clearly establish any levels of measurement being used e) Make sure that the conclusions/recommendations are clearly linked to the evaluation process. Pros: Cons: Maintaining academic rigor can be challenging. If a thorough literature review is not conducted, research papers of this type can often end up merely repeating previous studies with no new value added. This is a relatively easy research design to understand. Many successful ACSC papers have employed this method.

Typical organization: Front matter (title page, disclaimer, table of contents, abstract, etc.) Introduction/the research question Background/literature review Methodology/explanation of evaluation criteria used Results of evaluation Analysis of results Recommendations Conclusions Back matter (appendices, endnotes, bibliography, etc.)

The Tongue and Quill provides additional information on different types of research frameworks (pages 348-353).9 Available at http://www.epublishing.af.mil./shared/media/epubs/afh33-337.pdf. Created by Dr. John T. Ackerman and Dr. Matthew C. Stafford, ACSC/DL [Updated by Dr. Kathleen A. Mahoney-Norris, ACSC/DL, December 2010] Sources: Air Force Handbook (AFH) 33-337, The Tongue and Quill, 1 August 2004. Online. Creswell, John W. Research Design: Qualitative, Quantitative, and Mixed Methods Approaches, 3rd edition. Thousand Oaks, CA.: Sage Publication, 2009.

Flyvbjerg, Bent. Five Misunderstandings about Case Study Research. Qualitative Inquiry, 12, no. 2, April 2006: 219-245. Johnson, Janet B. and Richard A. Joslyn. Political Science Research Methods 3rd edition. Washington D.C.: Congressional Quarterly Press, 1995. Scearce, Diana and Katherine Fulton. What If? The Art of Scenario Thinking for Non-profits. Emeryville, CA: Global Business Network, 2004. On-line: Internet. Available at http://www.gbn.com/articles/pdfs/GBN_What%20If.pdf. Schwartz, Peter. The Art of the Long View: Planning for the Future in an Uncertain World. New York: Double Day, 1991.
1 2

Creswell, Research Design, 153-175. Johnson and Joslyn, Political Science Research Methods, 111-148. 3 Schwartz, The Art of the Long View, 101; Scearce and Fulton, What If?, 27. 4 Schwartz, The Art of the Long View. 5 Ibid., 241-247. 6 Ibid., 247-248. 7 Scearce and Fulton, What If?, 24-33. 8 Flyvbjerg, Five Misunderstandings about Case Study Research, 221-224. 9 AFH 33-337, The Tongue and Quill, 348-353.