Course Information

My Contact Details Dr. Zdravko Botev Room 2056, Red Centre Email botev@unsw.edu.au Consultation hours: Monday at 1pm, RC-2056

. .. ..

. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. ..

. ..

. . . .. .. ..

Week 1 Lecture 1 - Outline.

Introduction to regression models The model building process The simple linear regression model Estimation of parameters by least squares Examples to illustrate!

. .. ..

. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. ..

. ..

. . . .. .. ..

Week 1 Lecture 1 - Linear models introduction

It is a common task when performing statistical analysis to solve problems involving sets of variables. It is often the case that one knows that there exists an inherent relationship between the variables. This course will explore statistical questions relating to general linear relationships between variables Regression Models. Regression modelling: describe the behavior of some variable of interest (the response) in terms of a collection of related variables (the predictors).

. .. ..

. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. ..

. ..

. . . .. .. ..

Week 1 Lecture 1 - Linear models

Example 1: Industrial application. It is known that tar content in an outlet stream in a chemical process is related to the inlet temperature. Question: How can one develop a method of prediction procedure for estimating tar content for various levels of the inlet temperature given experimental data ? In practice - likely that many example runs (experiments) with the same inlet temperature will produce dierent levels of tar content. What is a reasonable model ?

. .. ..

. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. ..

. ..

. . . .. .. ..

Week 1 Lecture 1 - Linear models

Example 2: Retail - Housing market. Consider houses for sale in the same neighborhood with the same size block of land, in practice they will not all have the same sale price. Question: How can one develop a method of prediction procedure for estimating sale price based on the size of the block of land ?

. .. ..

. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. ..

. ..

. . . .. .. ..

Week 1 Lecture 1 - Linear models

Example 3: Daily maximum temperature forecasts at Sydney Airport. Consider the daily maximum temperature at Sydney airport, how does max temp. relate to forecasts of meteorological variables from a physical model (based on equations of motion of a uid). Question: How can one develop a method of prediction procedure for estimating maximum temperature based on the meteorological models ?

. .. ..

. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. ..

. ..

. . . .. .. ..

Week 1 Lecture 1 - Linear models

Tar content, House prices and Max daily temp. are natural dependent variables responses ! Inlet temperature, Land size and forecasts of meteorological variables are natural independent variables regressors! This course studies the relationship between the response and the regressors in a linear framework, e.g Y = + x where is the intercept and is the slope. (1)

. .. ..

. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. ..

. ..

. . . .. .. ..

Week 1 Lecture 1 - Linear models

In an ideal world this relationship would be exact one unique response for a given regressor input value! This is not the case in real world models models approximate the real world process (not exact)! Statistical Model Statistical model relates response to physical model predictions, allows better predictions and quantication of uncertainty about the response. Regression analysis deals with nding the best relationship between responses and regressors in a particular class of models.

. .. ..

. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. ..

. ..

. . . .. .. ..

Week 1 Lecture 1 - Linear Models

Predictors are often under the control of an experimenter. Having the predictors controlled in a well designed experiment may be important for making inferences about the eect of the predictors on the response. We assume in this course that the predictors are controlled in a designed experiment or at least accurately measured.
Those interested in design of experiments: (see e.g. Design and analysis of experiments Hinkelmann and Kempthorne, vol 1.)

. .. ..

. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. ..

. ..

. . . .. .. ..

Week 1 Lecture 1 -The Model Building Process

Why do we build statistical models? We build statistical models in order to make decisions. What is a good model? Not such a simple question to answer, several aspects to consider! (t, predictive performance, parsimony, interpretability...) All models are wrong, but some are useful (attributed to George Box)

. .. ..

. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. ..

. ..

. . . .. .. ..

Week 1 Lecture 1 -The Model Building Process

More examples: Patients with a certain condition are assigned to various courses of treatment: which treatment is most eective? Predicting the future value of a share market index based on information available to us now. Comparing scientic theories: which is best supported by the data?

. .. ..

. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. ..

. ..

. . . .. .. ..

Week 1 Lecture 1 - The Model Building Process

Qualitative description of a model Response = Signal + Noise Y = + x + where represents the noise. Signal: variation in response explained in terms of predictors, and a small number of unknown parameters (systematic part of the model). Noise: residual variation unexplained in the systematic part of the model (this part of the model may also be described in terms of some unknown parameters). Statistical model provides a reduction of a possibly large and complex data set to a small number of parameters
. .. .. . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . . . . .. .. ..

(2)

Week 1 Lecture 1 - Diagramatic Explanation

Figure : Probability and Statistics for Engineers and Scientists, Walpole et. al., 8th ed.

. .. ..

. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. ..

. ..

. . . .. .. ..

Week 1 Lecture 1 -The Model Building Process

A good statistical model provides a reduction of the data meaningful for making a decision. Model ts the data well if the systematic part of the model describes much of the variation in the response (low noise). However tting the data well may require a model with a large number of parameters. Parsimony :(small number of parameters) is desirable, as a smaller number of parameters provides a greater reduction of the data, may be more useful for making a decision.

. .. ..

. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. ..

. ..

. . . .. .. ..

Week 1 Lecture 1 - Diagramatic Explanation 2

Figure : Probability and Statistics for Engineers and Scientists, Walpole et. al., 8th ed.
. .. .. . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . . . . .. .. ..

Week 1 Lecture 1 -The Model Building Process

Cycle between tentative model formulation, estimation of parameters, model criticism. Must manage a trade o in selecting a good model between goodness of t and complexity of the model. Model should provide a reduction of the data useful for making the decisions which motivated the building of the model. In this course we consider only the class of linear models.
Simplest example: simple linear regression model.

Illustrate process of model selection, estimation and model criticism for this case rst. Extend ideas to the general linear model.
. .. .. . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . . . . .. .. ..

Week 1 Lecture 1 -Simple Linear Regression Model

Model response variable in terms of a single predictor. y1 , ..., yn values of the response variable. x1 , ..., xn corresponding values of the predictor variable. Each yi is a realization of a random variable Yi |Xi = xi . E (Yi |Xi = xi ) = 0 + 1 xi , where 0 and 1 are unknown parameters (intercept and slope terms respectively). Var (Yi |Xi = xi ) = 2 and responses are uncorrelated.

. .. ..

. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. ..

. ..

. . . .. .. ..

Week 1 Lecture 1 - Simple Linear Regression Model

Another way of writing the simple linear regression model: yi = 0 + 1 xi + i where the i s are uncorrelated errors with mean zero and variance 2 . Normality not required in derivation of estimators of unknown parameters. Normality of errors i N (0, 2 ) typically assumed allows one to derive condence intervals and hypothesis tests explicitly. Assumption of normality allows stronger conclusions about optimality of estimators of unknown parameters.

. .. ..

. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. ..

. ..

. . . .. .. ..

Week 1 Lecture 1 - Estimation of Parameters

Aim: Choose parameters so that predictions of tted model (values of systematic component of the tted model) match responses well in some overall sense. Minimize goodness of t criterion. Let y i = 0 + 1 xi (tted values). Possibilities for measuring goodness of t:
n (yi y i )2 i =1 n i =1

|yi y i |.

First of these criteria (least squares criterion) is computationally tractable and gives estimators with optimal . . . . . . . . . . . . . . . . properties in case of normal errors. . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .

. . . .. .. ..

Week 1 Lecture 1 - Least Squares and the Fitted Model

Aim: obtain estimates b0 and b1 of 0 and 1 respectively. Allows for computation of predictions from tted line y = b0 + b1 x , as well as goodness of t diagnostics. Residual - the error in the t of the model ei = yi yi . (small residuals can be a sign of good t).

Figure : Probability and Statistics for Engineers and Scientists, Walpole et.
al., 8th ed.
. .. .. . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . . . . .. .. ..

Week 1 Lecture 1 - Estimation of Parameters

Least squares estimators b0 , b1 . Choose b0 , b1 to minimize S=
n i =1 n i =1

(yi y i )2 =

(yi 0 1 xi )2

with respect to 0 and 1 . Theorem 1: The parameters minimizing the sum of squared residuals are given by, n (xi x ) (yi y ) Sxy b1 = i =1 = n 2 Sxx ) i =1 (xi x b0 = y b1 x .
. .. .. . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . . . . .. .. ..

(3)

Week 1 Lecture 1 - Estimation of Parameters

Sketch of proof: Derivation of least squares estimators 2 1. Finding arg minb0 ,b1 n i =1 (yi b0 b1 xi )
(SSE ) b0 (SSE ) b1

= 2 = 2

n n

i =1 (yi i =1 (yi

b0 b1 xi ) b0 b1 xi )xi

2. Set each partial derivatives of SSE with respect to b0 and b1 to zero normal equations.
n n nb0 + b1 i =1 x i = i =1 yi n n n b0 i =1 xi + b1 i =1 xi2 = i =1 xi yi

3. Solve the simultaneous equation for the expressions!

. .. ..

. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. ..

. ..

. . . .. .. ..

Week 1 Lecture 1 - Example

Figure : Scatter plot showing Zinc concentrations in plants (y-axis) and lake sediment samples (x-axis)

. .. ..

. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. ..

. ..

. . . .. .. ..

Example: Zinc concentrations in plants

15 i =1 15 i =1

yi = 808.4,

15 i =1 15 i =1

xi = 1205.2

xi2 = 101228.6,

xi yi = 68716.78

Sxy = 68716.78

(808.4).(1205.2) = 3764.5 15 1205.22 = 4394.8 15

Sxx = 101228.6

b1 = 3764.5/4394.8 = 0.857 b0 = 808.4/15 0.857 1205.2/15 = 14.9.

. .. .. . . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. .. . .. . . . . .. .. ..

Learning Expectations.

Understand the quantitative and qualitative need for statistical models. Understand the model building process (at least in the linear case). Be able to formulate the simple linear model and evaluate least squares estimates of the parameters. Be able to derive the least squares estimates of the parameters.

. .. ..

. . . . . . . . . . . . . . .. .. .. .. .. .. .. .. .. .. .. .. .. ..

. ..

. . . .. .. ..