Regression: Linear Regression Nonlinear Regression

Numerical Method-Sudarmaji Interpolation
1. Linear Regression
2. Nonlinear regression
REGRESSION

What is regression? Given n data points
) , ( , ... ), , ( ), , ( 2 2 1 1 n n y x y x y x
best fit
) (x f y =
to the data. The best fit is generally based on
minimizing the sum of the square of the residuals, r S
Residual at a point is
) ( i i i x f y = c
=
=
n
i
i i r x f y S
1
2
)) ( (
) , ( 1 1 y x
) , ( n n y x
) (x f y =
Figure. Basic model for regression
Sum of the square of the residuals
.
) , ( , ... ), , ( ), , ( 2 2 1 1 n n y x y x y x

Given n data points best fit
x a a y
1 0
+ =
to the data.
Does minimizing
=
n
i
i
1
c work as a criterion, where
) ( 1 0 i i i x a a y + = c
x
i i i
x a a y
1 0
= c
1 1
, y x
2 2
, y x
3 3
, y x
n n
y x ,
i i
y x ,
i i i
x a a y
1 0
= c
y
Figure. Linear regression of y vs. x data showing residuals at a typical point, x
i
.
x y
2.0 4.0
3.0 6.0
2.0 6.0
3.0 8.0
Example: Given the data points (2,4), (3,6), (2,6) and (3,8), best fit
the data to a straight line using Criterion#1
Figure. Data points for y vs. x data.
Table. Data Points
0
2
4
6
8
10
0 1 2 3 4
x
y
0
4
1
=
= i
i
c
x y y
predicted
= y - y
predicted

2.0 4.0 4.0 0.0
3.0 6.0 8.0 -2.0
2.0 6.0 4.0 2.0
3.0 8.0 8.0 0.0
Table. Residuals at each point for
regression model y = 4x 4.
Figure. Regression curve for y=4x-4, y vs. x data
0
2
4
6
8
10
0 1 2 3 4
x
y
Using y=4x-4 as the regression curve
) (
bx
ae y =
) (
b
ax y =
|
.
|
\
|
+
=
x b
ax
y
Some popular nonlinear regression models:
1. Exponential model:
2. Power model:
3. Saturation growth model:
4. Polynomial model:
) ( 1 0
m
mx a ... x a a y + + + =

Given n data points
) , ( , ... ), , ( ), , ( 2 2 1 1 n n y x y x y x
best fit
) (x f y =
to the data, where
) (x f
is a nonlinear function of
x
.
Figure. Nonlinear regression model for discrete y vs. x data
) (x f y =
) , (
n n
y x
) , (
1 1
y x
) , (
2 2
y x
) , (
i i
y x
) (
i i
x f y
) , ( , ... ), , ( ), , (
2 2 1 1 n n
y x y x y x
Given best fit
bx
ae y =
to the data.
Figure. Exponential model of nonlinear regression for y vs. x data
bx
ae y =
) , (
n n
y x
) , (
1 1
y x
) , (
2 2
y x
) , (
i i
y x
i
bx
i
ae y
( )
=
=
n
i
bx
i
r
i
ae y S
1
2
The sum of the square of the residuals is defined as
Differentiate with respect to a and b
( )( ) 0 2
1
= =
c
c
=
i i
bx
n
i
bx
i
r
e ae y
a
S
( )( ) 0 2
1
= =
c
c
=
i i
bx
i
n
i
bx
i
r
e ax ae y
b
S
Rewriting the equations, we obtain
0
1
2
1
=
= =
n
i
bx
n
i
bx
i
i i
e a e y
0
1
2
1
=
= =
n
i
bx
i
n
i
bx
i i
i i
e x a e x y
Substituting a back into the previous equation
0
1
2
1
2
1
1
=
=
=
=
=
n
i
bx
i
n
i
bx
bx
n
i
i
bx
i
n
i
i
i
i
i
i
e x
e
e y
e x y
The constant b can be found through numerical
methods such as bisection method.
=
=
=
n
i
bx
n
i
bx
i
i
i
e
e y
a
1
2
1
Solving the first equation for a yields
t(hrs) 0 1 3 5 7 9
1.000 0.891 0.708 0.562 0.447 0.355
Many patients get concerned when a test involves injection of a
radioactive material. For example for scanning a gallbladder, a
few drops of Technetium-99m isotope is used. Half of the
techritium-99m would be gone in about 6 hours. It, however,
takes about 24 hours for the radiation levels to reach what we
are exposed to in day-to-day activities. Below is given the
relative intensity of radiation as a function of time.
Table. Relative intensity of radiation as a function of time.

Find:
a) The value of the regression constants
A
and
b) The half-life of Technium-99m
c) Radiation intensity after 24 hours
The relative intensity is related to time by the equation
t
Ae
=
The value of is found by solving the nonlinear equation
( ) 0
1
2
1
2
1
1
=
=
=
=
=
=
n
i
t
i
n
i
t
n
i
t
i
t
i
n
i
i
i
i
i
i
e t
e
e
e t f

=
=
=
n
i
t
n
i
t
i
i
i
e
e
A
1
2
1
t
Ae
=
( ) 0
1
2
1
2
1
1
=
=
=
=
=
=
n
i
t
i
n
i
t
n
i
t
i
t
i
n
i
i
i
i
i
i
e t
e
e
e t f

t (hrs) 0 1 3 5 7 9

1.000 0.891 0.708 0.562 0.447 0.355
( ) 0
1
2
1
2
1
1
=
=
=
=
=
=
n
i
t
i
n
i
t
n
i
t
i
t
i
n
i
i
i
i
i
i
e t
e
e
e t f

t=[0 1 3 5 7 9]
gamma=[1 0.891 0.708 0.562 0.447 0.355]
syms lamda
sum1=sum(gamma.*t.*exp(lamda*t));
sum2=sum(gamma.*exp(lamda*t));
sum3=sum(exp(2*lamda*t));
sum4=sum(t.*exp(2*lamda*t));
f=sum1-sum2/sum3*sum4;
1151 . 0 =
The value of A can now be calculated
=
=
=
6
1
2
6
1
i
t
i
t
i
i
i
e
e
A
9998 . 0 =
The exponential regression model then is
t
e
1151 . 0
9998 . 0

=
t
e
1151 . 0
9998 . 0

=
The relative intensity of radiation after 24 hours
( ) 24 1151 . 0
9998 . 0

= e
2
10 3160 . 6

=
This result implies that only
% 317 . 6 100
9998 . 0
10 316 . 6
2
=

radioactive intensity is left after 24 hours.
What is the half-life of Technetium 99m
isotope?
Write a program in the language of your
choice to find the constants of the model.
Compare the constants of this regression
model with the one where the data is
transformed.
What if the model was ?
t
e
=

THE END

http://numericalmethods.eng.usf.edu
) , ( , ... ), , ( ), , ( 2 2 1 1 n n y x y x y x
Given best fit
m
m
x a ... x a a y + + + =
1 0
) 2 ( s n m to a given data set.
Figure. Polynomial model for nonlinear regression of y vs. x data
m
m
x a x a a y + + + =
1 0
) , (
n n
y x
) , (
1 1
y x
) , (
2 2
y x
) , (
i i
y x
) (
i i
x f y
The residual at each data point is given by
m
i m i i i
x a x a a y E = . . .
1 0
The sum of the square of the residuals then is
( )
=
=
=
=
n
i
m
i m i i
n
i
i r
x a x a a y
E S
1
2
1 0
1
2
. . .
To find the constants of the polynomial model, we set the derivatives
with respect to
i
a
where
( )
( )
( ) 0 ) ( . . . . 2
0 ) ( . . . . 2
0 ) 1 ( . . . . 2
1
1 0
1
1 0
1
1
1 0
0
= =
c
c
= =
c
c
= =
c
c
=
=
=
m
i
n
i
m
i m i i
m
r
i
n
i
m
i m i i
r
n
i
m
i m i i
r
x x a x a a y
a
S
x x a x a a y
a
S
x a x a a y
a
S

, , 1 m i = equal to zero.
These equations in matrix form are given by
(
(
(
(
(
(
(
(
(
(
=
(
(
(
(
(
(
(
(
(
(
(
(
(
(
|
.
|
\
|
|
.
|
\
|
|
.
|
\
|
|
.
|
\
|
|
.
|
\
|
|
.
|
\
|
|
.
|
\
|
|
.
|
\
|

=
=
=
= =
+
=
=
+
= =
= =
n
i
i
m
i
n
i
i i
n
i
i
m
n
i
m
i
n
i
m
i
n
i
m
i
n
i
m
i
n
i
i
n
i
i
n
i
m
i
n
i
i
y x
y x
y
a
a
a
x x x
x x x
x x n
1
1
1
1
0
1
2
1
1
1
1
1
1
2
1
1 1
. . .
. . .
. . .
. . . . . . . . . . .
. . .
. . .
The above equations are then solved for
m
a a a , , ,
1 0

Temperature, T
(
o
F)
Coefficient of
thermal
expansion,
(in/in/
o
F)
80 6.4710
6

40 6.2410
6

40 5.7210
6

120 5.0910
6

200 4.3010
6

280 3.3310
6

340 2.4510
6

Regress the thermal expansion coefficient vs. temperature data to
a second order polynomial.
1.00E-06
2.00E-06
3.00E-06
4.00E-06
5.00E-06
6.00E-06
7.00E-06
-400 -300 -200 -100 0 100 200
Temperature,
o
F
T
h
e
r
m
a
l

e
x
p
a
n
s
i
o
n

c
o
e
f
f
i
c
i
e
n
t
,

(
i
n
/
i
n
/
o
F
)
Table. Data points for
temperature vs
Figure. Data points for thermal expansion coefficient vs
temperature.

2
2 1 0
T a T a a + + =
We are to fit the data to the polynomial regression model
(
(
(
(
(
(
(
=
(
(
(
(
(
(
(
(
(
(
|
.
|
\
|
|
.
|
\
|
|
.
|
\
|
|
.
|
\
|
|
.
|
\
|
|
.
|
\
|
|
.
|
\
|
|
.
|
\
|

=
=
=
= = =
= = =
= =
n
i
i i
n
i
i i
n
i
i
n
i
i
n
i
i
n
i
i
n
i
i
n
i
i
n
i
i
n
i
i
n
i
i
T
T
a
a
a
T T T
T T T
T T n
1
2
1
1
2
1
0
1
4
1
3
1
2
1
3
1
2
1
1
2
1
o
o
o
The coefficients
2 1 0
, a ,a a
are found by differentiating the sum of the
square of the residuals with respect to each variable and setting the
values equal to zero to obtain
The necessary summations are as follows
Temperature, T
(
o
F)
Coefficient of thermal
expansion, (in/in/
o
F)
80 6.4710
6

40 6.2410
6

40 5.7210
6

120 5.0910
6

200 4.3010
6

280 3.3310
6

340 2.4510
6

Table. Data points for temperature vs. 5
7
1
2
10 5580 . 2 =
= i
i
T
7
7
1
3
10 0472 . 7 =
= i
i
T
10
7
1
4
10 1363 . 2
=
=
i
i
T
5
7
1
10 3600 . 3

=
=
i
i
o
3
7
1
10 6978 . 2

=
=
i
i i
To
1
7
1
2
10 5013 . 8

=
=
i
i i
T o
(
(
(

=
(
(
(
(
(
(
1
3
5
2
1
0
10 7 5
7 5 2
5 2
10 5013 . 8
10 6978 . 2
10 3600 . 3
10 1363 . 2 10 0472 . 7 10 5800 . 2
10 0472 . 7 10 5800 . 2 10 600 . 8
10 5800 . 2 10 6000 . 8 0000 . 7
a
a
a
Using these summations, we can now calculate
2 1 0
, a ,a a
Solving the above system of simultaneous linear equations we have
(
(
(
=
(
(
(
11
9
6
2
1
0
10 2218 . 1
10 2782 . 6
10 0217 . 6
a
a
a
The polynomial regression model is then
2 11 9 6
2
2 1 0
T 10 1.2218 T 10 6.2782 10 6.0217

+ =
+ + = T a T a a
To find the constants of many nonlinear models, it results in solving
simultaneous nonlinear equations. For mathematical convenience,
some of the data for such models can be transformed. For example,
the data for an exponential model can be transformed.
As shown in the previous example, many chemical and physical processes
are governed by the equation,
bx
ae y =
Taking the natural log of both sides yields,
bx a y + = ln ln
Let y z ln = and
a a ln
0
=
(implying)
o
a
e a =
with b a =
1
We now have a linear regression model where
x a a z
1 0
+ =
Using linear model regression methods,
_
1
_
0
1
2
1
2
1 1 1
1
x a z a
x x n
z x z x n
a
n
i
n
i
i i
n
i
i
n
i
n
i
i i i
=
|
.
|
\
|
=

= =
= = =
Once
1
, a a
o
are found, the original constants of the model are found as
0
1
a
e a
a b
=
=
t(hrs) 0 1 3 5 7 9
1.000 0.891 0.708 0.562 0.447 0.355
Many patients get concerned when a test involves injection of a radioactive
material. For example for scanning a gallbladder, a few drops of Technetium-
99m isotope is used. Half of the technetium-99m would be gone in about 6
hours. It, however, takes about 24 hours for the radiation levels to reach what
we are exposed to in day-to-day activities. Below is given the relative intensity
of radiation as a function of time.
Table. Relative intensity of radiation as a function
of time
0
0.5
1
0 5 10
R
e
l
a
t
i
v
e

i
n
t
e
n
s
i
t
y

o
f

r
a
d
i
a
t
i
o
n
,

Time t, (hours)
Figure. Data points of relative radiation intensity
vs. time
Find:
a) The value of the regression constants
A
and
b) The half-life of Technium-99m
c) Radiation intensity after 24 hours
The relative intensity is related to time by the equation
t
Ae
=
t
Ae
=
Exponential model given as,
( ) ( ) t A + = ln ln
Assuming ln = z ,
( ) A a
o
ln =
and =
1
a we obtain
t a a z
1 0
+ =
This is a linear relationship between
z
and t
Using this linear relationship, we can calculate
1 0
, a a

= =
= = =
|
.
|
\
|
=
n
i
n
i
i
n
i
i
n
i
n
i
i i i
t t n
z t z t n
a
1
2
1
2
1
1 1 1
1
and
t a z a
1 0
=
where
1
a =
0
a
e A =
1
2
3
4
5
6
0
1
3
5
7
9
1
0.891
0.708
0.562
0.447
0.355
0.00000
0.11541
0.34531
0.57625
0.80520
1.0356
0.0000
0.11541
1.0359
2.8813
5.6364
9.3207
0.0000
1.0000
9.0000
25.000
49.000
81.000
25.000 2.8778 18.990 165.00
Summations for data linearization are as follows
Table. Summation data for linearization of data model
i i
t
i
i i
z ln =
i i
z t
2
i
t
With 6 = n
000 . 25
6
1
=
=
i
i
t
=
=
6
1
8778 . 2
i
i
z
=
=
6
1
990 . 18
i
i i
z t
00 . 165
6
1
2
=
= i
i
t
Calculating
1 0
, a a
( ) ( )( )
( ) ( )
2
1
25 00 . 165 6
8778 . 2 25 990 . 18 6

= a 11505 . 0 =
( )
6
25
11505 . 0
6
8778 . 2
0

= a
4
10 6150 . 2

=
Since
( ) A a ln
0
=
0
a
e A =
4
10 6150 . 2

= e 99974 . 0 =
11505 . 0
1
= = a
also
Resulting model is
t
e
11505 . 0
99974 . 0

=
0
0.5
1
0 5 10
Time, t (hrs)
Relative
Intensity
of
Radiation,
t
e
11505 . 0
99974 . 0

=
Figure. Relative intensity of radiation as a function of
temperature using linearization of data model.
The regression formula is then
t
e
11505 . 0
99974 . 0

=
b) Half life of Technetium 99 is when
0
2
1
=
=
t

( )
( )
( )
hours . t
. t .
. e
e . e .
t .
. t .
0248 6
5 0 ln 11505 0
5 0
99974 0
2
1
99974 0
11508 0
0 11505 0 11505 0
=
=
=
=

c) The relative intensity of radiation after 24 hours is then
( ) 24 11505 . 0
99974 . 0

= e
063200 . 0 =
This implies that only
% 3216 . 6 100
99983 . 0
10 3200 . 6
2
=

of the radioactive
material is left after 24 hours.
Comparison of exponential model with and without data linearization:
With data linearization
(Example 3)
Without data linearization
(Example 1)
A 0.99974 0.99983
0.11505 0.11508
Half-Life (hrs) 6.0248 6.0232
Relative intensity
after 24 hrs.
6.320010
2
6.316010
2

Table. Comparison for exponential model with and without data
linearization.
The values are very similar so data linearization was suitable to
find the constants of the nonlinear exponential model in this
case.


Regression: Linear Regression Nonlinear Regression

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Regression: Linear Regression Nonlinear Regression

Uploaded by

Copyright:

Available Formats

Numerical Method-Sudarmaji Interpolation

Numerical Method-Sudarmaji Interpolation

Numerical Method-Sudarmaji Interpolation

You might also like