Fall 2004 Masters Comprehensive / PhD Preliminary Exam September 17, 2004 Morning Session Measurement Measurement 1.

(a) Briefly define three types of validity as discussed in the Standards for educational and psychological testing. (b) Provide an example of a situation where construct validity methods might be applied. (c) Describe research and/or statistical methods typically used in determining or demonstrating construct validity. (d) Discuss limitations or problems commonly associated with applications of construct validity.

Measurement 2. (a) When assessing the quality of test items, it is common to calculate the relationship between getting the item correct or incorrect and the total test score. Name two correlations that satisfy this purpose. (b) Compare and contrast the assumptions of these two correlations. (c) Discuss the advantages and disadvantages of using each coefficient. (d) Under what conditions is one coefficient definitely to be preferred to the other?

Measurement 3. (a) Produce and label a graph representing an item characteristic curve resulting from the 3-parameter logistic item response theory model. (b) Briefly define each parameter in the model. (c) Produce a second graphical representation for which the values of all 3 parameters have increased in relationship to your initial graph. (d) Provide a practical interpretation of an increase in the value for each parameter.

Measurement 4. (a) Distinguish between a universe of admissible observations associated with a Generalizability study and a universe of generalization associated with a Decision study in Generalizability theory. (b) Suppose an investigator conducts a p x t x r G study using np=100 persons, nt=5 essay items, and nr = 6 raters. What variance components may be estimated by such a design? (c) What would be the interpretation of the Generalizability coefficient from this design and how would it be calculated? (You do not have to actually calculate the coefficient.)

Research Methods Research Methods 1. The analysis below represents the relationship between 3 predictors (X1, X2, and X3) and a criterion variable (Y).
- - Correlation Coefficients Y X1 X2 1.0000 .1965 .1620 .1965 1.0000 .6484 .1620 .6484 1.0000 -.1389 -.0468 .0038 - X3 -.1389 -.0468 .0038 1.0000

Y X1 X2 X3

------------------ Variables in the Equation -----------------B SE B Beta T Sig T X2 .069157 .046243 .067390 1.495 .1352 X3 -.644750 .167312 -.132339 -3.854 .0001 X1 .184873 .056871 .146643 3.251 .0012 (Constant) 10.825611 .582502 18.585 .0000

In the context of the results above: (a) (b) (c) (d) Discuss the difference in meaning and interpretation of B (raw) and (Beta; standardized) regression weights. Discuss how the equation weights above relate back to the correlation matrix. Discuss the predictor contributions in this equation. How would you expect the equation weights for X2 and X3 to change if X1 is removed from the analysis?

Research Methods 2. A study was made of a sample of EDMS 645 students. The students in the study were all 1st semester graduate students and were distributed across 7 University departments. Students were randomly distributed to 3 groups. Over the first 5 weeks of a semester, group 1 was given easy problem assignments, group 2 received moderately difficult problem assignments, and group 3 received very difficult assignments. The groups were then compared on their self-ratings of the usefulness of statistics. No attempt was made to assess or control for student differences in math background. Some students received special statistics tutoring during the 1st 5 weeks of the semester unbeknownst to the researcher. (a) (b) (c) (d) Discuss potential threats to internal validity posed by this study. Describe two ways to deal with the threats discussed in a. Discuss potential threats to external validity posed by this study. Describe two ways to deal with the threats discussed in c.

Fall 2004 Masters Comprehensive / PhD Preliminary Exam September 17, 2004 Afternoon Session Research Methods, continued

Research Methods 3. A research design may include one or both of these random procedures: (i) random sampling from some specified existing population and (ii) random assignment of experimental participants to treatment levels within the experiment. Discuss how the use of each of these procedures contributes to the meaningfulness and interpretation of the research outcome. Research Methods 4. Suppose that a researcher is doing a study of the effects of Artistic Training on 4th grade students in Virginia. Ten schools volunteer and she gets agreements to have the Art teachers provide the training that she has prepared. A test of Art Knowledge is also developed and will be administered at the beginning and at the end of the art training. She also knows that she should use a control group, but the Virginia schools will not let her randomly select schools. So she selects control schools that are as similar to the 10 experimental schools as possible, using a set of 3 variables (average SES of each school, State-test average for each school, and percentage of non-native English speaking students in each school). She will test all the 4th grade students in those "control/comparable" schools at the same time that she tests the students from the experimental schools.

(a) (b) (c)

(d)

What are the risks with this approach to finding a comparison group? What does she gain from getting a comparison group in this way, compared to not having a comparison group of students at all? What statistical analysis should she use for the research design described? Defend your choice of analysis and discuss the advantages and disadvantages of the analysis you recommend. Discuss the advantages and disadvantages of the research design described compared to using randomly selected schools as the experimental and control schools (i.e., full random selection).

Statistics Statistics 1. Suppose that a directional z test is used with =.05 to test Ho: 50 against H1: > 50, where = 10 and n = 25. (a) (b) (c) Sketch the sampling distributions of the mean for = 50 and for = 55 on a common axis. Label appropriate regions for Level of Significance; 1 ; ; and Power. Explain how the sketch would change if the sample size was increased.

Statistics 2. There is a lot of discussion going on in psychology about the usefulness of classical hypothesis testing as an indication of the importance of results. If one study rejects the null hypothesis at the .01 level and another study rejects at the .05 level, is the first indicative of a stronger effect? (a) Discuss the concern with determining importance from classical hypothesis testing? How do sample size and standard error contribute to this discussion? (b) What indices besides statistical significance could a statistician utilize to describe the magnitude of an effect?

Statistics 3. In experimental applications, a two-way ANOVA with one experimental factor and one blocking variable and an ANCOVA (or ordered regression analysis) with a covariate or quantitative control variable are both ways of reducing experimental error. (a) (b) Contrast the research situations in which each would be appropriate Explain how each procedure leads to error reduction

Statistics 4. A researcher wonders if students reported stress levels (measured by a stress inventory) varies over the semester. She randomly selects 15 students from each of the 4 main campuses of the University of Maryland and asks them to complete a stress inventory during Weeks 1, 7, and 14 of the semester. Higher scores indicate greater stress. The SPSS output from this study is below.
b Mauchly's Test of Sphericity

Measure: MEASURE_1 Epsilon Within Subjects Effect Mauchly's W TIME .075 Approx. Chi-Square 46.685 df 2 Sig. .000 Greenhous e-Geisser .519
a

Huynh-Feldt Lower-bound .523 .500

Tests the null hypothesis that the error covariance matrix of the orthonormalized transformed dependent variables is Tests of Within-Subjects Effects proportional to an identity matrix. a. May be used to adjust the degrees of freedom for the averaged tests of significance. Corrected tests are displayed in the Measure: MEASURE_1 Tests of Within-Subjects Effects table. Type III Sum b. Source of Squares df Mean Square F Sig. Design: TIME Intercept Sphericity Assumed 93.733 2 46.867 3.652 .035 Within Subjects Greenhouse-Geisser Design: TIME 93.733 1.039 90.230 3.652 .069

Error(TIME)

Huynh-Feldt Lower-bound Sphericity Assumed Greenhouse-Geisser Huynh-Feldt Lower-bound

93.733 93.733 487.600 487.600 487.600 487.600

1.045 1.000 38 19.738 19.863 19.000

89.663 93.733 12.832 24.704 24.549 25.663

3.652 3.652

.069 .071

(a)

(b)

Should he researcher infer a statistically significant (at =.05) TIME effect based on these results or not? Explain your answer in light of all relevant portions of the output, and specifically mention what df and pvalue(s) you base your answer upon. In the context of this study, briefly describe what the sphericity assumption is about and what adjustments are made if it is violated and why.