y

y y y y

y y y y y y y y y y

Validity: refers to the likely truth of an assertion; how well a study procedure or a measure does what it is supposed to do o Internal Validity: truthfulness of the assertion that the observed effect is due to the independent variable (that the IV caused changes in DV). o External Validity: generalizability of study findings to other populations, places and times Meta Analysis (Research Synthesis): the process of using statistical analysis to combine results of different research studies testing similar or the same hypotheses Theory: a formalized set of concepts (constructs) that organizes observations and inferences; also predicts and explains phenomenon Testable Theory: Makes specific predictions that we can evaluate using a structured scientific approach Construct or Concept: an abstract, unmeasured aspect of people, events or things o Theories relate how two concepts or constructs are linked. For example, overcrowding and crime are associated with one another. o Theories specify the suspected nature of the relationship among the constructs o Cause (Exogenous Construct) Effect (Endogenous Construct) Variables: concrete measures that operationalize constructs Independent variable: measures the exogenous construct; it is the cause in our theory Dependent variable: measures the endogenous construct; it is the effect in our theory Indirect Causal Relationship: (Exogenous Construct) (Intervening Construct) (Endogenous Construct) Reciprocal Causation: (Construct) (Construct) Conceptual Hypothesis: prediction about relationships among constructs that make up our theory Operational Hypothesis: prediction about relationship among variables in our theory Parsimony or Efficiency: a theory with parsimony employs the fewest constructs and linkages necessary to explain the events of interest Generality: a more general theory is useful because it explains a broader range of findings and goes beyond the observed to make predictions about circumstances not observed Causal Inference: o Covariation: if A causes B, then A & B should be associated with one another o Temporal Precedence: Cause should precede the effect in time. A should occur before B o Rule out plausible rival explanations for the observed association. Measurement: assigning values (e.g. numbers) to variables that represent attributes or properties of people, places or things o Levels of Measurement:  Nominal Scale: lowest level of measurement; groups observations with the same attribute together

 

Ordinal: groups observations and ranks or orders the observations Interval Scale: groups, orders, and assumes equal distance between adjacent categories Ratio Scale: groups, orders, equal distance between adjacent categories, and there is a true zero point

Error:  Random Error (Er): unsystematic, chance factors that affect measurement. y Example: Repeat Assessments of Blood Pressure  Non-random Error: systematic error affecting measurement. There are 2 sources of non-random error: y Bias: systematically affects all measurements in the same way y Error from Measuring the Wrong Construct  X = T + E(r) + B + E(w) y Observed Score = True Score + [Random Error + (Bias + Wrong Construct Error)] [Error], (Non-random Error)

y o

Properties of Measurement:  Reliability: the extent to which a measurement yields the same result on repeated trials, i.e., the degree to which observed scores are free from errors of measurement y based on the amount of random error relative to the observed score y estimated in different situations by calculating reliability coefficients y Estimate Reliability: o Reliability Coefficient: Kappa Statistic ( )  Appropriate when measures are nominal or ordinal  Interpreted as a chance corrected percentage of agreement o Reliability Coefficient: Intra-class Correlation (ric)  Appropriate when two or more measures are interval or ratio (sometimes ordinal)  Measures agreement between two or more measures o Reliability Coefficient: Pearson Correlation coefficient (r):  Appropriate when two measures are Interval or Ratio Scales (sometimes ordinal)  Measures linear association between two measures  Correlation ranges from -1.0 to +1.0 with y -1.0: perfect negative linear association

y y y y y y

0: no linear association +1.0: perfect positive linear association < 0.5 low reliability 0.5 - 0.7 modest reliability > 0.7 high reliability

Assess Reliability: o Test-retest Reliability: same test given to same subjects at 2 points in time  Measured by correlation between 2 administrations of same test  Example: CES-D measured at Time 1 and Time 2; weight example  Problems: y Time Interval: elapsed time allows for an actual change in the true score (real development) y Memory: people tend to remember responses and answer questions in the same manner o Parallel Test Reliability: same people given two similar or equal tests that use different items (usually at different times)  Solves memory problem  Still threat of actual change in true score (development) o Within Test Consistency: uses only a single test given at a single point in time.  Coefficient Alpha ( ): estimates how well test would correlate with an equal length test, composed of similar items, taken under the same conditions  = N x rmean / 1 + rmean(N-1)  N = # items  rmean = average inter-item correlation  Size of Alpha ( ) depends on: y Size of inter-item correlations y Number of items  Can improve alpha by: y Adding more items y Deleting items with poor inter-item correlations o Interrater Reliability: agreement between scores from two observers of same phenomenon  Examples: 2 physicians diagnosing same sample of patients; 2 observers rating # aggressive acts by children on playground

Implications of Low Reliability: o Attenuation: association between measures with low reliability will be attenuated - underestimated o Regression toward the mean: on average, high scores on first test will get lower on second test; extremely low scores on first will improve on second assessment, on average. y Improve Reliability: o Standardization: reduces random error associated with measurement  Train Interviewers: ask questions in same manner using same wording  Control the environment: use same room; same time of day Validity: concerns the relationship between construct and variable; how well the variable accurately reflects the designed construct y based on degree to which the measure accurately reflects the desired attribute or construct how large is E(w) relative to the true score? y is more theoretically oriented are we measuring what we think we re measuring y a measure is valid when it measures what it is supposed to y The amount of non-random error defined as Bias (B) and Error from measuring the wrong construct (E(w)) determine the amount of validity y Components of Validity: o Bias: systematic under- or over-estimation of measurement  Impacts validity because measure or test is not accurately reflecting the actual measure o E(w): error from measuring wrong construct -- error attributed to measuring something other than what we think we re measuring y Estimate Validity: o Criterion Validity: established by demonstrating that observed measure correlates highly with criterion measure.  Criterion: an existing measure known to be an adequate, valid measure of target construct  Types of Criterion Validity: y Predictive Validity: observed measure obtained before measurement of the criterion measure o Examples: SAT scores predicting college success measured by GPA (criterion measure) y

y y

Concurrent Validity: observed measure and criterion obtained at the same time o Example: Medical setting, use of noninvasive measure to predict results of a gold-standard or criterion that is obtained invasively.  Advantage: easily interpreted validity coefficient (e.g. correlation coefficient)  Drawbacks: y Assumes a criterion exists may not for complex measures like self- esteem or depression. y Assumes the criterion is valid. o Content Validity (Face): established by inspecting content of measures; usually based on expert judgment  Depends on subjective judgment, no validity coefficient (r) produced o Construct Validity: how well the observed test or measure reflects the target construct.  Assess relations among items of a test : use statistical procedures (Factor Analysis or Structural Equation Modeling) y Identifies the number of constructs measured by the set of items. y A measure that is composed of a single construct is called uni-dimensional  Relation of test to other measures of same construct y Different from criterion validity -- no agreed upon criterion; makes no assumptions about validity of measures y High correlations with other similar measures are evidence that measure is valid  Test that links among different constructs used in a theory conform to predictions y Explores how well measure being validated conforms to theory y Assumes theory is correct Census: measures entire population Surveys : based on a part or sample of population o Cross- sectional : collect data at one point in time y

Longitudinal : take place over time, two or more data collections, measure change over time  Panel Survey: data collected at different times from same people y Advantage: temporal order established y Potential Problems: o Attrition: loss of respondents over time. o Reactivity (Pretest sensitization): experience of 1st interview causes change in behavior by the 2nd interview. o Changes in the measuring procedures over time  Trend Survey: new sample of people is drawn at each time y avoid problems of: o attrition o reactivity y Disadvantage: not possible to establish temporal order Medium: method used to gather data  Face-to-face: usually most desirable y Observe non-verbal behaviors y CAPI (Computer assisted personal interview) y Interview people without telephones or addresses y Expensive  Internet: web-based data collection y advantages of economy, speed, and wide geographical coverage y Issues to Consider: o Human Subject protection o Identity of respondent o Privacy o Quality of day o How sample defined o Representativeness of Sample  Telephone Medium: costs about 1/2 of a face to face interview y More controlled y More than 90% of people in U.S. in urban areas have phones y Random Digit Dialing (RDD) y Disadvantages: o Respondents often remain anonymous o Calls screened using answering machines o Phone interviews may yield different answers than face to face interviews  Respondents refuse to answer or give false answers more easily to phone interviewer

Response Style (a form of Response Bias): tendency to answer items in manner unrelated to item content Sample: Subset taken from larger population. Goal: sample is representative of population  Sampling frame: partial or incomplete enumeration (listing) of population from which sample is drawn  Sampling Unit: what or who is studied y Possible Units : o individual people o groups  Ecological Fallacy: making assertions about individuals as unit of analysis based on the study of groups or other aggregations y Durkheim a famous 19th century sociologist o Protestant Regions for suicide; who is really committing suicide?  Types of Samples: y Probability Sample: every element of population has a known probability of selection o Random Sample: every element of population has equal chance of selection & selection of any one element doesn t affect selection of another (they are independent) o Systematic Sample: every nth element is sampled from an existing list or sampling frame o Stratified random sample: population is divided into strata or subgroups, a random sample taken from each strata  ensures that certain subgroups of population are adequately represented in sample o Advantages of Probability Sampling:  More likely to be representative  Results more likely to generalize to population  Probability Sampling maximizes : y External Validity: generalizability of study findings to other populations, places and times o Disadvantages of Probability Sampling:  Often difficult to implement to implement without a sampling frame y Non-probability Sample: elements have unknown chance of selection o Convenience Sample: subject select themselves o Purposive Sample: researchers choose subjects because of certain characteristics o Quota Sample: sample created to match a predetermined demographic profile

Advantages of Non-Probability Sampling: Disadvantages Non-Probability Sampling:  Samples not necessarily representative  Results may not generalize Sources of Error: y Random error: Unsystematic, unpredictable, just as likely to over- or under-estimate true value o Interviewer Effects: haphazard mistakes (reduce reliability of measures)  Standardization minimizes Interviewer Effects o Sampling Error: unavoidable, chance variation among different samples drawn from same population  Large Sampling Error leads to imprecise survey estimates  For probability samples, sampling Error is measured by Standard Error of Mean (SEM) y SEM: Standard Deviation/ n o n= sample size o Standard Deviation= variability of measured phenomenon  Increasing sample size (n) decreases sampling error y Increases precision of survey estimates  Plan for large enough sample size to insure acceptable levels of precision  y Non-random error (bias): Systematic over- or under-estimation of true value o Sample Bias: occurs when sample doesn t adequately represent population  Probability samples minimize sample bias o Completion Rates: extent to which sample is reached and persuaded to cooperate  when low, can lead to: y Non-observation Bias: responders may differ from non-responders  # Participating/# Contacted  # Participating/# Sampled (#contacted + #not contacted) o Interviewer Effects  Interviewer asks questions differently of some people  Personal mannerisms or characteristics of interviewer influences response o o

y y

 Standardization minimizes Interviewer Effects Experimental Research: research design in which the IV is fixed by a manipulation or natural occurence o Experimental Designs:  Establish whether association exists  Often determine temporal order  Differ in ability to control rival explanations  (independent variable goes before dependent variable)  Types of Experimental Designs: y True Experiments o Random assignment: method of placing subjects in different groups defined by I.V. so each subject has equal chance of being in any group  avoids systematic subject differences between groups.  minimizes group threats to internal validity  Confounding Variable: Uncontrolled variable affecting outcome; is associated with both the I.V. and the D.V. o Differently treated groups (e.g., include a control group)  Control group: Group of subjects who don t receive experimental treatment and serve as basis of comparison for other groups (defined by I.V.) y minimize time threats to internal validity y Quasi Experiments o don t use random assignment Correlational Research: research design in which the I.V. is measured rather than fixed by an intervention (manipulation or natural occurrence) Cross-sectional, correlational designs: o Primary threat to internal validity is  Reverse Causation: association in which the causal direction is opposite to that hypothesized