Lecture 2 Experimental Research

A teacher is never a giver of truth; he is a guide, a pointer to the truth that each student must find for himself. Lecture 2 Compulsory Readings Nunan. D. 1992. Research Methods in Language Learning. Cambridge: CUP (P. 24 - 50) Selinger, H.W., and E. Shohamy. 1989. Second Language Research Methods. Oxford: OUP (P. 135 - 152) Brown, J.D. 1988. Understanding Research in Second Language Learning : A Teachers Guide to Statistics and Research Design. New York: CUP Hatch, E. and A. Lazaraton, 1991. The Research Mannual: Design and Statistics for Applied Linguistics. New York: Newburry House What is an experimental research?

Experimental research is carried out to explore the strength of relationship between variables An experimental research is: - a controlled look at nature. (McDonough J.& McDonough S., 1997) - an attempt to maintain over all factors that may affect the result of an experiment. (James P., 1997) - the effects of specified and controlled treatments given to subjects usually formed into groups. (Selinger & Shohamy, 2000)

Moderator variable: affect the relationship between the independent & dependent variables by modifying the effect of the intervening variable In a questionnaire study, interest might focus on the way two different kinds of pupil, e.g. girls and boys answered the questions. Here sex would be a moderator variable. Control variable: the variables that are not measured in a particular study must be held constant, neutralized, or eliminated, so they will not have a biasing effect on the other variables Extraneous variable: are those factors in the research environment which may have an effect on the dependent variable but which are not controlled Confounding variables (CV):

Often difficult to separate out variables from one another satisfactorily.

The researcher would have a difficult time sorting out the difference between a group-work task and a group-work seating position. There are two variables, which might be important, are so far confounded. Confounding variables are undesirable features of experiment.

Type of scale Nominal scale: the nominal scale places things into named categories Ordinal scale: the ordinal places things in order Interval scale: the interval scale uses equal-sized units of measurement & shows distances between subjects performances Ratio scale: the ratio scale employs equal intervals & begins at a true zero point

Inferential statistics: The statistics which may be inferred from sample to population T-test: the test to compare two means (2 groups) F-test: the test to compare more than two means (more than 2 groups) ANOVA: Analysis Of Variance, a procedure to compare more than two means

Chi-square: calculate the different between the expected frequencies and compare them with the actual frequencies Correlation: a statistical test to estimate the degree (strength) of association between two variables Control group: the group under normal situation

Experimental group: the group under experimental condition

Kinds of experimental research TRUE- EXPERIMENT

QUASI- EXPERIMENT PRE- EXPERIMENT (Nunan 1992)

True-experiment

Random selection of subjects Use of control groups Random assignments to control and experimental groups Measurements: pre- and post treatment tests Highly controlled, highly selective

Advantages: - Greater internal validity - Causal claims can be investigated Disadvantages: - Less external validity - Not very feasible

Quasi-experiment

No random selection of subjects Use of control groups No random assignments to control and experimental groups Measurements: pre- and posttests

Advantages: - Greater external validity - Much more feasible given time and logistical constraints Disadvantages: - Not as many variables controlled

Pre-experiment

No random selection of subjects No control group Measurements: (pre- and posttests) Pilot study, one-shot case study,one group study...

Advantages: - Very practical - Set the stage for further research

Disadvantages: - Lower validity

Components of an experimental research Research questions

Literature review Hypothesis & focused question Variables

Validity Internal validity: did the experimental treatment make the difference in this specific instance rather than other extraneous variables?

External validity: to what populations, settings, treatment variables, and measurement variables can this observed effect be generalized? The Logic of Statistical Inference Mean X The average of a set of scores The central tendency of the scores

The Logic of Statistical Inference Mode: The score that occurs most frequently in a set of scores. Median: the middle point in a distribution, the midway between the two middle scores

The central tendency of the scores

Standard Deviation SD

Important measure of dispersion

Estimating: 1) deducting the mean from each individual score 2) Squaring the resulting figures to get rid of minus signs 3) adding these together and dividing by the total scores minus one 4) Obtaining the square root of the figure SD

Variance

Standard Error SE

The standard deviation of sample means: placing a single sample mean in relation to the population mean Estimating: Dividing SD of a single sample by the square root of the number of observations in the sample (N)

SE = SD : N Statistics have shown that mortality increases perceptibly in the military during wartime (?)

Questions for Tutorial 2

Question: Find a report of an experimental study in applied linguistics and give your comments on the following checklist: I. Introduction: Is the framework for the study clear? A. Literature review: can you tell where the study fits in? 1. Is the background or rationale provided? 2. Is the relationship to previous research clear? B. Statement of purpose: can you tell where the study is heading? Are any of the following included? 1. Purpose 2. Research question 3. Research hypothesis II. Method: Is the study replicable? A. Subjects 1. Is the description of participants adequate? 2. Is the method of selection clear? B. Materials 1. Is there a description of tests, questionnaires, rating scales, and so forth? 2. Do the variables represent reasonable operational definitions of the underlying constructs or characteristics involved? 3. Is there a description of any equipment (when applicable)? C. Procedures 1. Is there a description of the preparation of material, administration, scoring and so on? 2. Is there a description of the conditions during study? D. Analyses 1. Is there a description of the arrangement and grouping of the data? 2. Are the statistical tests listed in order of use? III. Results A. Are all the statistical tests previously listed represented as results ? B. Is there a prose explanation (optional)? IV. Discussion/Conclusion A. Is the original research question, or questions, answered ? B. Is there an explanation of why the results were as they were? 1. If the conclusion is based on previous research, is it well supported and reasoned? 2. If the conclusion is speculative, is it qualified as such and well reasoned? C. Are suggestions for further research provided? V. References, Notes, or Footnotes A. Are all the references cited in the text included ?

B. Are nay pertinent references missing? VI. Appendixes A. Are they necessary? B. Are they complete?