tomhen@gmail.

com
10 2015

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . :
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . :
1.1

4
4

1.2

. . . . . . . . . . . . . . . . . . . . . . . . . . . . :
1.1.1
). . . . . . . . . . . . . . . . . . . . . . :(Splines

5
9

1.3

. . . . . . . . . . . . . . . . . . . . . . . . . :Natural Cubic Spline

1.2.1
) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . :(Splines

11
11

1.4

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . :R
. . . . . . . . . . . . . . . . . . . . . . . :RN
1.4.1

21
21

Spline Interpolation . . . . . . . . . . . . . . . . . . . . :RN

Spline Filters . . . . . . . . . . . . . . . . . . . . . . . . :RN

22
24

. . . . . . . . . . . . . . . . . . . . . . . . . . . :Scattered Data Interpolation

1.5
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . :

25
27

2.1

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . :R

27

2.2

. . . . . . . . :Spline-Interpolation
2.1.1
N
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . :R

28
30

2.2.1
2.2.2

. . . . . . . . . . . . . . . . . . :Monte-Carlo
. . . . . . . . . . . . . :Importance Sampling

30
32

2.2.3
2.2.4

. . . . . . . . . . . . . . . . . . . . . . :Inverse Transform Sampling

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . :RN

32
34

2.2.5
2.2.6

Rn . . . . . . . . . . . . . . . :Rejection Sampling
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . :

35
35

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . :
). . . . . . . . . . . . . . . . . . . . . . . . . . . . . :(Bisection Method
3.1

36
36

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . :Newton-Raphson
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . :
3.2.1

36
36

Newton-Raphson . . . . . . . . . . . . . . . . . . . . . . . . :RN
3.2.2
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . :

38
38

LU . . . . . . . . . . . . . . . . . . . :

39

LU " . . . . . . . . . . . . . . . . . . :
LU . . . . . . . . . . . . . . . . . . :A

39
42

4.2

Cholesky . . . . . . . . . . . . . . . :
Cholesky . . . . . . . . . . . . . . . . . . . :
4.2.1

43
44

4.3

. . . . . . . . . . . . . . . . . . . . . . . . :
. . . . . . . . . . . . . . . . :
4.3.1

44
45

4.3.2
4.3.3

. . . . . . . . . . . . . :
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . :Jacobi

45
47

4.3.4
4.3.5

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . :Gauss-Seidel
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . :Richardson

48
49

4.3.6
4.3.7

. . . . . . . . . . . :Steepest-Descent
. . . . . . . . . . . . . . . . . . . . . . . . . . :Conjugate-Gradient

51
54

1.4.2
1.4.3
2

3.2

4.1

4.1.1
4.1.2

). . . . . . . . . . . . . . . . . . . . . . . . . . . :(Eigen-Decomposition
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . :Power-Method
5.1

55
55

5.2
5.3

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . :Inverse Power-Method
) Lanczos (. . . . . . . . . . . . . . . . . . . . . . . . . :

57
57

5.4

) Jacobi Iteration (. . . . . . . . . . . . . . . . . . . . :

58

raananf@cs.huji.ac.il

1:
1.1 :
m
m + 1 f : Rn R {(xi , yi := f (xi ))}i=0 ) f (x
x . } x Conv {yi , ..., yn
.x
} / Conv {y1 , ..., yn
1.2 f ,
) (.
1.3 :Order of Interpolation
m
f : R R {(xi , f (xi ))}i=0 .m
1.4 m m 1 .
m

{(xi , f (xi ))}i=0 m " :

"" :Fitting f :
f (xi ; c0 , ..., ck ) = f (xi ) := yi 0 i m

) c0 , ..., ck" (k m )(.

"" :Evaluation c0 , ..., ck " x" ) .f (x; c0 , ..., ck

1.5 f (xi ) = yi ) i (
.Strict Interpolation R
.
1.6 .

1.1

1.7 :
.cj
" ) f (x; c0 , ..., ck ,c0 , ..., ck :
j (x) cj

k
X

= ) f (x; c0 , ..., ck

j=0

{j }j=0 .
:
def k
z}|{ X
cj j (xi ) = yi
= ) 0 i m : f (xi ; c0 , ..., ck
j=0

A ~c = ~y A ) (m + 1) (k + 1 ) .Aij = j (xi
1.8 , , '.

1.1

1.1.1 :
1.9 :
) k(



, 1, x, x2 , ...., xk j (x) = xj } .j {0, ..., k :
cj xji = f (xi ) = yi

k
X
j=0

0 i m :

1.10 :
,Aij = xji
A k = m
Q
) .det (A) = i>j (xi xj det (A) 6= 0" xi 6= xj i 6= j
{xi , f (xi )}ni=0 .
A ,
) (.
:
c0 , ..., cm
:
Pk
j=0 xji cj = yi .


.O m3
: ) O (m .


: O m2.

m
m
m + 1 . m {xi , yi }i=0 '
:
m Y
X
) (x xk
yj
(x
) j xk
j=0
j6=k

= )P (x

0 ) ' (x

2

.O m

1.11 '
) .O (2m"
.A~c = ~y
:
Pm
p (x) = i=0 ci xi ) (
.
round off error
.machine-error ,1 + = 1 floating-point
108 . 1016 double
" " .
1.12 32 a [10, 10] ,f = a 10b
"" b .
f1 = a1 10b1 f2 = a2 10b2 b1 b2 f1 + f2 b2
f2 ) f1 22
10b 10 .(a
5

1.1

:
. .
:Neville
{(xi , yi )}m

.
machine-error
i=0
xi . (j < i) Pj,j+1,...,i
j k i ,Pj,j+1,...,i (xk ) = yk
P0,...,m .
: Pi (x) = yi ,0 i m .
: j < i 1 i j 1
Pj,j+1,...,i1 (xk ) = yk .j k i 1
: Pj,j+1,...,ii Pj+1,j+2,...,i Pj,j+1,...,i
i j :
)(x xj ) Pj+1,j+2,...,i (x) + (xi x) Pj,j+1,...,ii (x
)(x xj ) + (xi x
)(x xj ) Pj+1,j+2,...,i (x) + (xi x) Pj,j+1,...,ii (x
=
xi xj

= )Pj,j+1,...,i (x

j k i .Pj,j+1,...,i (xk ) = yk
: m m .
m
x
/ {xi }m
: {(xi , yi )}i=0 i=0
) P (x) := P0,...,m (x
. .


: ) O m2 m m (.

: y0 , ..., ym ).O (m
: machine errors .

1.1

:
1.13 :
m

) P (x m {(xi , f (xi ))}i=0 f m + 1

m
] [a, b .{xi }i=0 ] x [a, b ] [a, b :
)w(x

z
|}
{
m
)(m+1
Y
f
)(

) (x xi
= )R (x) = f (x) P (x
(m + 1)! i=0
: P R (xi ) = 0 .0 i m ) S (x :
)w(x

|}
{
z
m
Y
)(x xi ) S (x
= )R (x
i=0

) R(x x0 , ..., xm ) S (x . x
)w(x
[a, b] \ {xi }m F : [a, b] R :
i=0

)F (z) = f (z) P (z) w (z) S (x

F m + 2 x0 , ..., xm , x m + 1 ] .[a, b

Rolle m + 1 ] [a, b F . Rolle

) F (m+1 ] [a, b ] [a, b . :
=0

{ |} z
!)F (m+1) (z) = f (m+1) (z) P (m+1) (z) S (x) w(m+1) (z) = f (m+1) (z) S (x) (m + 1

!)0 = F (m+1) () = f (m+1) () S (x) (m + 1

)f (m+1) (
!)(m + 1

)f (m+1) (
!)(m + 1

= )= S (x

)= R (x) = w (x) S (x) = w (x

.
1.14

)(m+1

f ] [a, b .

1.15 ] [a, b a = x0 .b = xm
m

1.16 {(xi , f (xi ))}i=0 ] [x0 , xm

xi+1 xi = ) 0 i m ( .
] x [x0 , xm ] [x0 , xm :

 m
m
) f (m+1) (
Y
f (m+1) ()  Y


) |R (x)| =  (x xi
)  =  (x xi
!) (m + 1
 

(m
+
!)1
i=0
i=0
7

1.1



f (m+1)  K ] [x0 .xm ] x [x0 , xm:


m

Y
K


) |R (x)|  (x xi
!) (m + 1

i=0

] x [x0 , xm:



m

Y


! (x xi ) m+1 m


i=0

x"" . :
! m+1 m
m+1
=K
K
!)(m + 1
m+1

K
m+1

xm x0
m

m+1

|)|R (x

= :
xm x0
m

|)|R (x

" ,K , xm x0 ,m
. :


f "" m + 1 f (m+1)  1 .
f .
" " " ) sin (5x m
5m1 .
" . < 1
1.17
.
" .
:
m .m 1
:
.1 .Machine-Error
.2

3

O m

2

.O m

.3
.
.

1.2

):(Splines

1.2

):(Splines
n

) {(xi , yi )}i=0" xi( Spline :

) m 2" m (.
m + 1 ) x0 , ..., xm xm+1 , ...., x2m+2 (
.m
.
1.18
0
,
. xn x
n
:
n
. m

) O (1
m
) m ).(O (1

.|R (x)| m+1 K
m m.
1.19 m m ,
,
.
: Spline .
n

1.20 {(xi , yi )}i=0

) ( . Spline
. Pi Pi+1
,xi :
Pi (xi ) = Pi+1 (xi ) = yi

Pi (xi ) = Pi+1 (xi ) = yi

yi :



) f (xi+1 ) f (xi
2
) = f (xi ) + o (xi+1 xi
) (xi+1 xi

.Pi (xi ) = Pi+1 (xi ) = 0

Pi (xi ) = Pi+1 (xi ) = yi:

.1 Pi (xi ) = Pi+1 (xi ) = yi yi.

.2

) f (xi+1 )f (xi
) (xi+1 xi

= ) .Pi (xi ) = Pi+1 (xi

.3 .Pi (xi ) = Pi+1 (xi ) = 0

1.2

):(Splines

)
( , " 4n 2 .
4n ) n n + 1 (.
1.21 Spline .Cubic Splines
:Spline
) (.

y0

c0
2
3
1 x0 x0 x0 c1 y1

1 x1 x21 x31
.. ..

..
.. .
= .
.
. . .
.. ..
1 xm x2m x3m
cm
ym

Cubic Spline xi , xi+1

:

c0 + c1 xi + c2 x2i + c3 x3i = Pi (xi ) = yi

c0 + c1 xi+1 + c2 x2i+1 + c3 x3i+1 = Pi (xi+1 ) = yi+1

c1 + 2c2 xi + 3c3 x2i = Pi (xi ) = yi

c1 + 2c2 xi+1 + 3c3 x2i+1 = Pi (xi+1 ) = yi+1

:

i
yi
c0
x3i1

i
x3i
c1 yi+1
2 i =
yi
c2
3xi1

ci3
3x2i
yi+1

x2i1
x2i
2xi1
2xi

xi1
xi
1
1

1
1

0
0

,4 4 n
.4n 4n
) O (n
.
1.22 x
Spline x"
).O (1
:

) Pi1 (xi ) = yi = Pi (xi

.

) Pi1 (xi ) = Pi (xi )" (.

3 .1
n + 1 n 3n 4
)) 3 (n 2 2 ( . 3n 4 3
n 3n 4 . :
3) m = 2 ( , n 3 .
10

1.3

) :(Splines

2) m = 1 (
3) m = 2 (.
" 3 (n 3) + 2 + 3 = 3n 4 .
.


.O n3 , Back-Substitution
) O (n ) ( ) ,O (n
).O (1
1.23
.
1.2.1 :Natural Cubic Spline
Spline {(xi , yi )}ni=0 :
0 i n : Pi (xi ) = yi = n+1 conditions
1 i n 1 : Pi (xi+1 ) = Pi+1 (xi+1 ) = n-1 conditions

0 i n 2 : Pi (xi+1 ) = Pi+1 (xi+1 ) = n-1 conditions

0 i n 2 : Pi (xi+1 ) = Pi+1
(xi+1 ) = n-1 condtions

,4n 2 P0 (x0 ) = 0 Pn (xn ) = 0

4n n , Natural
,Cubic Splines :

.
.
..
..

, ,
.
) (


.O n3 Back-Substitution ).O (n

1.3 ) :(Splines

Spline
Spline , Data .
,
:
{xi }ni=0 ) xi+1 xi = .(i
11

) :(Splines

1.3

x ) (
.
n on,k
j
zi+1
zij zij =
x
i=0,j=0

) j = 0, ..., k (.

n on,k
rij

{(xi , yi )}ni=0
n on,k
. zij \

i=0,j=1

i=0,j=1

.xi \ {(xi , yi )}i=0

n
] [x1 , xn {(x1 + a, y1 )}i=0
n
n {(xi , yi )}i=0 ] [x1 , xn
] .[x1 + a, xn + a o
n

xi + a, yi

i=0

] [x1 + a, xn + a

) y1 , ...yn .(y1 , ..., yn

mj
mj
{(xi+l , yi+l )}l=1 xi , .., xi+mj
mj
zij .rij
,{(xi+l , yi+l )}l=1
mj :

m j
1
xi
x2i

xi j
c0
) P (xi
yi
m
j
2
cj
yi+1 P (xi+1 ) 1 xi+1
x

x
i+1
i+1
1

= ..
.. =
.. .
..
.
.
. ..
.
mj
2
1 xi+mj xi+mj xi+mj
) P (xi+k
yi+mj
cjm
} | {z
|
{z
} } | {zj
y
~i

Mij

~
cj

y~i

1

Mi

= c~j rij

zij :

c0

 
  mj 
c1
. = P zij = rij
zij
..
}
c
} | {z

 2
zij

{z

yi+mj
} {z
y
~i

j
xi+m
j

| }

x2i+mj
{z
1
) (Mij

1
xi

1
x
  mj
i+1

..
zij
.
}
1 xi+mj
|

~yi := Aji y~i


j 1

Zij

c~j

m 1
x2i

xi j
yi
mj
2
yi+1
xi+1
xi+1

.. ..

.
.

zij

 2
zij

{z
Zij

zij

rij = 1
|

rij = Zij Mi

 j 1
i rij
Aji := Zij Mi .

Aji .i "
12

1.3

) :(Splines

j
i ~yi
j n rion
j
j
. ( zi ri
)
i=1

i ri1 , ..., rik :

yi
..
.

0
0
1 1
yi+m1
M

0
0
.
..
..
.
..
.
.

.

1
yi
k
0
M

k mk
1
zi
{zP
} ..
|}
P

.
m m
j

yi+mj
} {z
P
mj

m2
zi2
0
mj

{z

0
1

m1
zi1
0

1
1
ri
ri2 0

.. = .
. ..
0

rik
} | {z
~
ri

yi
).k1 = (k maxj mj )(maxj mj maxj mj )(maxj mj 1
.rij
1.24

n on
zij

i=1

) j

( )
( .
:
1.25 :(LTI) Linear Translation Invariant
rij .yi
j
.
rij = ri+k
, yi = yi+k

1.26
P, Q

{(xi , yi )}m
i=0

, yi )}m
i=0

{(xi + ).Q (x + ) = P (x

: Q, P m )P (xi ) = yi = Q (xi +
.0 i m ) ,R (x m :
)R (xi + ) = P (xi ) = Q (xi +
R, Q m m + 1 R Q x
) ,P (x + ) = R (x) = Q (x.
1.27
Aji .i
 j 1


: rij y1 , ..., yi+mj
Aji = Zij Mi
h i
) ,(1 mj 1 l mj Aji yi+l .rij
l

j
1 l mj h el
i i1 6= i2
Ai h .i i
l Rmj Aji1 = Aji1 el . Aji2 = Aji2 el :
l

 j 1
h i
~
el = Zij1 cji1
Aji1 = Aji1 el = Zij1 Mi1
l

13

1.3

) :(Splines

 j 1
~
cji1 = Mi1 Pel mj
el



(xi1 , 0) , ..., (xi1 +l , 1) , ..., xi1 +mj , 0

~
cji1 Zij1 Pel zij1
h i
 
Aji1 = Pel zij1
l

Qel



(xi2 , 0) , ..., (xi2 +l , 1) , ..., xi2 +mj , 0

 j 1
~
~
.cji2 = Mi2 Zij2 cji2 zij1 zij2
el
h i
 
Aji2 = Qel zij2
l

{xi }i=1 Qel Pel

) = (xi1 xi2 Qel
n




o
xji1 + , 0 , ..., xji1 +l + , 1 , ..., xji1 +mj + , 0

x ) .Pel (x) = Qel (x +

zij1 = zij2 + :
  h i


 
h i
Aji2 = Qel zij2 = Qel zij1 + = Pel zij1 = Aji1
l

" 1 l mj

Aji2

Aji1

Aji

.i

1.28 xi1 , ..., xi1 +mj xi2 , ..., xil +mj

zij1 zij2 .
: j rij } i {1, ..., Aj


" .rij = Aj ~yi ~yi = yi , ..., yi+mj xi , ..., xi+mj .
1.29 ):(j = 1, 2
} .{x0 , ..., xn

.zi2 = xi +

1.30 zn2 = xn + 2
. .
" . = 1 j = 1 i ri1 = yi j = 2 ,m2 = 1
)( " 12
zi2 = xi + 21 ) .(xi , yi ) , (xi+1 , yi+1 zi2
ri2 = 12 yi + 21 yi+1 .i i


 
A1 = 1 0 A2 = 12 21:
  1


1 0
ri
yi
=
1
1
ri2
yi+1
2
2
14

1.3

" :
 

yi
1 0
1
1
yi+1 1
2
2


 
1 0
yi
=
1
1
yi+1 2
2
2



) :(Splines

= ri1 = yi

1
1
yi + yi+1
2
2

= ri2

i ~r = A~y :

y1
1

r1
1 0
y2

r12 1 1

y3
1 2 2

r2
.
1 0
2
.
1
1
r2
.
2
2
=
.
..
.
.
..
.
.

rn1
1 0 yn2
yn1
1
1
rn2
2
2
yn
) k = 2 ( 1 ) (m1 = m2 = 1 A
.2n n = 2n n k A .(n k) n
k A Aj , ..., Ak

yi

1 .
rij = Z j M j
..
|
{z
}
yi+mj
Aj
:

y1
y2
y3
..
.
..
.

yn2

yn1
yn n1

1
A
..
.

nkn

Ak

0
A1

Ak

{z

1
1
A
r1
..
..
.
.
k
k
A
r1

1
0
r2

..
..
.
.

k
=
0
r2

.
.
..
..

0
r 1

n
.
.
..
..
k
0
rn nk1
|

~r = A~y A1 , ..., Ak Aj mj .j
rij r (i, j) := k (j 1) + i A
.rij = Ar(i,j), ~y Ar(i,j), Aj i

yi

rij = Aj ... = Ar(i,j), ~y

yi+mj

15

1.3

} {1, ..., k

mj
X
 

yi+k Aj k
= = Ar(i,j), ~y
k=0

) :(Splines

j } i {1, ..., n:

yi
j
j
ri = A ...

yi+mj

1.31 :
) f, g : Z R ( f g f g : Z R
n Z:
)f (k) g (n k

k=

= )(f g) (n

.Translation Invariance k ) g (n) = g (n + k:

)(f g) (n) = (f g) (n + k
) (Translations k : Z Z
k (n) = n + k .(f (g k )) = (f g) k
1.32 , , ,
RZ
.
(
1
i
=
0
= f = f .f RZ

0 i 6= 0
1.33 F
f, g RZ :
))F (f g) = F (f ) F (g) = f g = F 1 (F (f ) F (g
" ) F (f ) F (g F 1 .f g
:
n
X


 
)fkj yik = f j ~y (i
= yi+k Aj k
k=0

mj
X

rij

k=0

fkj :
mj k 0
otherwise

(

Aj
0

fkj

fkj " :
1.34
j
" .A
j  j
ri
j 
j
} j {1, ..., k r1 rn = f ~y f .Aj,

16

1.3

) :(Splines

1.35 :

y1
0
y2
1 0

..
..
.
.
yn
1 0

y1
y2

..
.
yn

.
1
2

1
2

..

1
1
r1
r21

.. =
.
rn1

1
2
1
2

2 1
r1
2
r22

.. =
.
rn2

0
1

1
1

r1

r21

.. =
.

rn1
0

1
2

zn1
, :
2 1 1

r1
2
2
1
1
r22
y1
2
2

y2

..

..
. =
..
.

.

1
1
2
rn1

2
2
yn
1
1
rn2
2
2

y1

y2

..
.

0 yn
1

0
..
.

..

0
1

.
1




0 1 0 0
:

y1
y
2
y
0
3
1
.
2
.
.
..
.
.
.
.
0 1

1
1
yn2

2
2

1 0 yn1
1
yn
0 2
{z
}
1
2

1
2

.
0
1
2

1
1
2

1
1
1
r1
2
r12
1
r2
2
r2
=
..
.

rn1

rn2
1
2

\
\  .
 ,
. 0 21 1 12 0

17

1.3

) :(Splines

1.36 :
C Rnn :

c0
cn1
c2
c1
c1
c0
cn1
c2

..
..
..

.
.
c
c
.
1
0

.
.
..
.. c

cn2
n1
cn1 cn2
c1
c0

.
) .(c0 , c1 , ..., cn1
A nk k nk nk
~y
. ~r = A~y :

1 0
y1
1
1
1

r1
y2

2 2

r12
1
0

y3

1
1
.
r2
2
2
.
2
.
r2
.
.
=
.

.
.
..
.
.
0
1

1
1
1
yn2
rn

2
2

rn2
1 0 yn1
1
1
yn
0
2



" 0 12 1 12 0 ~y
~y " "Zero Interpolation Zero-Upsampling ":
(
y 2i i is even

= ~yi
0
otherwise

 n .2n
 ~y

C 0 12 1 12 0
~y
~.C
y
=
~A
y


C 0 12 1 12 0
.
C , :

1
1 12 0 0
2
1
1

2 1 2 0
0
1 0

..

..

1 1
0 21 1 12
.
.

2 2

1
0

..
.

..

. 0 1 1
1
1

2
2

..
.

.
1
.
~y =

.
.
.
0
~y
.
2

.
.

..
.. 0
0 1

1
1

.
.

2
2
.
.

0
.
1 0

. . .
1
..
0 21
.. .. ..
2
. 12
|
{z
}
1
0 0 0
0 1
A
2
{z
}
|
C

18

1.3

) :(Splines

:Spline-Filters

j

} j {1, ..., k rn = f j ~y


r1j f j

.Aj,

k) k 1 (
Upsampling ~y n ~y

 n k



j
. j r1 rnj = f j ~y f j
) ( j C :
1
r1

1

..
y1
A
0
0
.
k
0
.

r1

..
A1
0
0
1
..

r2
.

.


.
Ak
.
A1
0
..
0

..
..
k
y2
k
=

.
0
A
0
.
r2

.
.

..
..
.
Ak
A1

..
.

.
.
r1

.
.
.
.
.
.

n
.
0
.
.
.
..
0
0 Ak nknk yk
nk1
|
{z
}
rnk nk1
C
 j 1
A1 , ..., Ak m1 , ..., mk "
.Aj := Z j M

" :

yi
2nd step
3rd step
4th step
1st step




{|}z
{|}z
{|}z
{|} z
j
=
Ar(i,j), ~y
=
)f j y (i
=
)Cj, ~y = f j ~y (i
= A ...
yi+mj

rij

f j j ,A:
 (
Aj k mj k 0
j
= fk
0
otherwise

f \ j .C

:
n on
rij"
j

i=1


j

rn = f

f . F :
 1

r1 r1k rn1 rnk = F ~y

F interleaving



F = f11 f12 f1k fn1 fnk

r1j

f 1 , ..., f :

" ~r = C ~y = F ~y F
.C

19

1.3

) :(Splines

1.37 m n m
m F y ) O (n m k) = O (n k F m k
y n k )
( . ) m = n (

upscale
O n2 FFT ).O (n log n
n

1.38 {(i, yi )}i=0

) i + 0.3 .(zi = i + 0.3 :

m 1
1
xi
x2i

xi j
yi
mj

yi+1

x2i+1
xi+1
 2
  mj 
1 xi+1

j
j
j
j

ri = 1 zi
..
.. ..
zi

zi
.

.
.
{z
}
|
mj
2
1 xi+mj xi+mj xi+mj
yi+mj
Zij
{z
} } | {z
|
y
~i

Mij

i:


 1

 1
yi
i

ri1 = 1 i + 0.3
yi+1
1 i+1
{z
}
|
A1

A1" i i = 0





  1 0 1



1 0
A1 = 1 0.3
= 1 0.3
= 0.7 0.3
1 1
1 1




, 0.7 0.3 0.3 0.7 i :


 
ri = 0.3 0.7 yi yi+1 = 0.3yi + 0.7yi+1

.i + 0.3
:Spline Filters

m
m
.m {xi , P (xi )}i=0 P m
Q .Q P"
Splines .Spline Filters

.m F
m



Pj (x) := xj | 0 j m F m + 1

) ,(Pj (xi )) F = Pj (zi ) Pj (xi ) Pj (zi

F . j = 0 Pj (x) = 1 :




g F = 1 0k1 1 1 0k1 1 1nk F = 1 1 1nk
:

=n

)n F (p

nk
X
p=1

{!

|}

)g (i p

nk
X
i=1

)F (p

F (p) = k

nk
X
p=1

= )F (p) g (i p

Pnk

p=1

nk X
nk
X
i=1 p=1

= (g F )i

nk
X

=1

i=1

. .
20

nk
X
i=1

= nk

1.4

1.4

:RN

:RN

n
~xi , f ~xi
RN
i=1
:

.~xi RN

n

X
) cj j (~xi
= yi := f ~xi
j=1

n .j : RN R
1.4.1 :RN
1.39 N m P : RN R
; i,j Z0 , ci R

xj i,j

N
Y

j=1

ci

k
X

= ) P (x1 , ..., xN

i=1

:
i,j

N
X
j=1

m := max

1ik

QN

1.40 i Z0 , j=1 xj j .RN

1.41 x2 + xy + xy 2 .3

m N
1.42
P
N "
1 , ..., N
Z0 j=1 j m


+m
+m
. Nm
= NN

RN
N ) (m n 1
) n (.
1.43 , .

(~xi )j i,j ; 1 i n

N
Y

j=1

cl

K
X

= ) yi = P (~xi

l=1

K = N +n1
P n 1 m
N




N +n1 .N, n > 1
N +n1 > n
n
N
N
overdetermined linear system ) (
. .

21

 :RN

1.4

1.44 N = 2 n = 3 2 m :

c1

c2

1 ~x1,1 ~x1,2 ~x1,1 ~x1,2 ~x21,1 ~x21,2

y1

c
3
1 ~x2,1 ~x2,2 ~x2,1 ~x2,2 ~x22,1 ~x22,2 = y2

c4
y3
1 ~x3,1 ~x3,2 ~x3,1 ~x3,2 321,1 ~x23,2
c5
c6
1.45 (x1 , ..., xN ) RN 1 i N Hi
n
xi xj .i 6= j {~xi , yi }i=1

Hj
) Hj (~xi

n
Y

j=1,j6=i

yi

n
X

=P (x1 , ..., xN ) :

i=1

) ' .(RN
.
1.46 Neville RN .N > 1
1.4.2

Spline Interpolation :RN

RN .

RN )
\ (.
1.47 RN ) (i1 , ..., iN 0 ij K
1 j N :


~xi1 ,...,iN = x1i1 ,...,iN , ..., xN
)) i1 ,...,iN = (f1 (i1 ) , ..., fN (iN
f1 , ..., fN : R R .

1.48:
.1 ij j .
.2 ~xi1 ,...,iN N 1 i1 , ..., iN K
N .
.3 fj (i) = a i + bj a R b RN
. fj (i) = aj i + bj
.4 " K N K N.
N

.{~xi1 ,...,iN , yi1 ,...,iN }i=1

1.49 2 j ,aj:
k~x21 ~x11 k = k(f1 (2) , f2 (1)) (f1 (1) , f2 (1))k = k(f1 (2) f1 (1) , 0)k
| = k((a1 2 + b1 ) (a1 1 + b1 ) , 0)k = k(a1 , 0)k = |a1

22

1.4

:RN

:
~x RN N
~x
.
:
""
.

N
m RN ) (m + 1 ) (.

.


R2 x1 , x2 R2 :
.1 X ) p1 , ..., pm+1 : R
R m ""

 ( x1 , x2 p1 , ..., pm+1 x1

.X
p1 x1 , ...., pm+1 x1


m

.2
x1 , pi x1 p : R R .Y
i=1




.3 .P x1 , x2 := p x2

RN m :
n

.1 D1 := {(~xi , yi )}i=1 ~x RN :1 i N
)( Di
.

N i

) Mi := (m + 1 pi,1 , ..., pi,Mi : R R i




)( pi,1 , ..., pi,Mi ~xi R . pi,1 ~xi , ...., pi,Mi ~xi



Mi
)(
Di+1 = ~xi , pi,j ~xi .i + 1
j=1

.2 N PN,1 : R R . P (~x) := PN,1 ~xN

,
R2 ) 32 = 9( .
3,3
{(~xij , yij )}i=1,j=1 :

0


2
x1

0

2
x1

0
1

0
1

2

2
x1
x1

0

0
y11

y12

y21

y21

y22

y23
y31
2
(x11 )

2 y32
) (x21
y33
(x31 )2

2

x1
0

2
1 x1
x211

2
x212 0 0

2
0
x213
2

) (x11
2
) (x21
2
) (x31

x11
x21
x31

1
1
1

x11
x21
x31

x211
x212
x213
2

1
i


2 2
1
x
1

) (x11
2
) (x21
2
) (x31
1
1
1

x11
x21
x31

1
1

0
23

P x1 , x2 = 1 x2

1.4

:RN

:


P x1 , x2 = U2 M21 U1 M11 ~y

M11 U1 U2 .
1.50 N m ) (m + 1)(N 1
N
N
( ) (m + 1) (m + 1

) 

.O N (m + 1)3 ) (m + 1)(N 1




O N mN , mN " .

1.51
" .
1.52 P .
:RN

n {(~xi , yi )}ni=1

P (~xi ) = yi" .


)) (O (n .O n3

Splines
O N mN m Spline " .n

1.4.3

Spline Filters :RN

1.53 N f, g : ZN R :
) f (i1 , ..., in ) g (z1 i1 , ..., zN iN

iN =

i1 =

=(f (x1 , ..., xN ) g (x1 , ..., xN )) (z1 , ..., zN ) :

1.54 f : ZN R f1 , ..., fN : Z R :
) fj (ij

N
Y

= ) f (i1 , ..., iN

j=1

Spline Filters {~xi1 ,...,iN , yi1 ,...,iN }i=1

oN,K
n
N
zij1 ,...,iN
i=1,j=1

. .

1.55 "
.
rij1 ,...,iN :
.1 .
24

 :Scattered Data Interpolation

1.5

.2 N .
.3 N N .
2 3 :

{|}z
)) fi (xi ) = g (f1 (x1 ) (x2 ) ... (xN

)) ( (x1 ) f2 (x2 ) (x3 ) ... (xN )) ... ( (x1 ) ... (xN 1 ) fN (xN

N
Y

i=1

g f (x1 , ..., xN ) = g

" rij1 ,...,iN g N yi1 ,...,iN

Up-Scaling ) ( )) ( (x1 ) ... (xN 1 ) fN (xN )) ,...,(f1 (x1 ) (x2 ) ... (xN
N )
fi : Z R .(i f1 , ..., fn
.f

, .
Spline-Filters :RN
RN .
fi : R R :
) fi (xi ) = (x1 ) ... (xi1 ) fi (xi ) (xi+1 ) ... (xN
.


Q
N
N )( ) ) f (x1 , ..., xN ) = i=1 fi (xi
Q
( .
f (x1 , ..., xN ) = N N .
N ) i=1 fi (xi
.

1.5

:Scattered Data Interpolation

RN
n
{(~xi , yi )}i=1 .
n
. {(x~i , yi )}i=1
x~i RN "" Radial
) Basis Functions .(Kernel Density Estimation
:
Pn
n
yi (k~xi ~xk) X
) (k~xi ~xk
yi Pn
=
(~x) := Pi=1
n
)xi ~xk
) (k~xi ~xk
~i=1 (k
i=1
| i=1 {z
}
=wi

x2

1
) (x) = e 2
.( (x) = |x|+
) i (~x) := (k~xi ~xk . :

.
25

1.5

:Scattered Data Interpolation

.
~x min yi (~x) max yi
.yi
~xi
)xk
~ wi = Pn(k ,yi
~xi
)xk
~i=1 (k

.
x2

(x) = e 2 fit
. ) (~x

q
. 
O N n1
q
. N n1

26

2 :
2.1

:R
b

f : R R ] [a, b . I = a f (x) dx
h = ba n
f Spline n
)
( . Spline
Newton-Cotes Spline . n
] [a, b xi = a + ih . 0 i n
:m = 1 Spline . Trapezoid-Method
. h
Spline :
) f (xi+1 ) f (xi
) (x xi
h

Pi (x) = f (xi ) +

:
x
i+1

(x xi ) dx

xi

xi+1


x
i+1


) f (xi+1 ) f (xi
Pi (x) dx = f (xi ) x  +
h
xi


xi+1
f (xi+1 ) f (xi ) x2
xi x
= f (xi ) h +
h
2
xi



f (xi+1 ) f (xi ) x2i+1 x2i
) f (xi+1 ) f (xi
(xi ) h + f (xi ) h +
h
h
2

=h
z
|}
{

f (xi+1 ) f (xi )
(xi+1 + xi ) (xi+1 xi )
= (f (xi+1 ) f (xi )) (xi ) + hf (xi ) +

h
2

) (f (xi+1 ) f (xi )) (xi+1 + xi

= [(f (xi+1 ) f (xi )) xi + hf (xi )] +
 2

) (xi+1 + xi
xi
)) = hf (xi ) + (f (xi+1 ) f (xi
2
h

{ |} z
xi+1 xi
h
)) = hf (xi ) + (f (xi+1 ) f (xi
)) = hf (xi ) + (f (xi+1 ) f (xi
2
2
h
h
)) = (2f (xi ) + f (xi+1 ) f (xi )) = (f (xi+1 ) + f (xi
2
2
" :
n1
hX
]) [f (xi+1 ) + f (xi
2 i=0

x
i+1

= Pi (x) dx

n1
X

i=0 x
i

f (x) dx

b
a

:m = 2 " n :
n

2 1
h X
f (x) dx
)) (f (x2i ) + 4f (x2i+1 ) + f (x2i+2
3 i=0

27

b
a

xi

:R

2.1

2.1 m = 1 h2 ) f (xi ) f (xi+1 m = 2

h
h3 , 4h ) ,f (x2i ) , f (x2i+1 ) , f (x2i+2 m = 1 h
3 , 3
m = 2 .2h" m m h
. m > 8
.
2.1.1 :Spline-Interpolation
Spline-Interpolation :m
 xi+m

x
xi+m
xi+m


i+1


m
 =Ei :
f (x) dx
P (x) dx
= |f (x) Pi (x)| dx
|Ri (x)| dx


xi

xi

xi

xi

Pi ] [xi , xi+m m .xi , ..., xi+m f

m + 1 m + 1 ] [a, b i :




)Ki := max f (m+1) (
] [xi ,xi+m

Ki

m+1

|Ri (x)| h ] x [xi , xi+m:

xi+m

|Ri (x)| dx = h m hm+1 Ki = mhm+2 Ki

Eim

xi
n

m ) n ( K := max Ki:



 a



n
n
x
x
xi+m

 m

 m
1
1  xi+m
a
i+1
i+1
X
X






m






P (x) dx
f (x) dx
 = E :=  f (x) dx P (x) dx
 P (x) dx
f (x) dx


  i=0


 i=0
xi

xi

xi

K = (b a) K hm+1 = O hm+1

xi

m+2

ba
n

n
mhm+2 K = nhm+2 K = n
m

Eim

n
m 1

i=0

) h (n .Spline
:
2.2 :
h
f : R R ][h, h . h f (x) dx = 0
x+
f x ] [x , x + . x f (x) dx = 0

f : R R 4 ] [h, h ] x [h, h:


x3
x2
f (0) + f (3) (0) + o x4
2
6
:

)(1

)(2

o x4 dx

h
h

x dx +

)f (3
x dx +
6

f (x) = f (0) + xf (0) +

)f (x
xdx +
2

)1dx + f (0

)f (x) dx = f (0

h3
2h3
f (0) + 0 = 2hf (0) + f (0) + o h5
= 2hf (0) + 0 +
6
3
28

h
h

:R

2.1

:
3

h
h
f (0) + f (3) (0) + o h4
2
6
2
3

)(h
)(h) (3
f (h) = f (0) + hf (0) +
f (0) +
f (0) + o h4
2
6
:




)f (h) + f (h) 2f (0
+ o h4
= )f (h) + f (h) = 2f (0) + h2 f (0) + o h4 = f (0
2
 h



)h3 f (h) + f (h) 2f (0
h3
+ o h4
= 2hf (0) + f (0) = 2hf (0) +
2
3
3
h




6
h
2
h
= hf (0) + (f (h) + f (h)) hf (0) + o h4 = (f (h) + f (h) + 4f (0)) + o h4
3
3
3
3
) (2 :

f (h) = f (0) + hf (0) +

h




h3
f (0) + o h5 = (f (h) + f (h) + 4f (0)) + o h4 + o h5
6
3


h
(f (h) + f (h) + 8f (0)) + o h5
3

f (x) dx = 2hf (0) +

2h ] [xi , xi+1
. xi+12xi :





xi + xi+1
f (xi ) + f (xi+1 ) + 4f
+ o h5
2

xi+1 xi
2

=h

x
i+1

h
= f (x) dx
3

xi

Spline ) (:
h
)) (f (x2i ) + 4f (x2i+1 ) + f (x2i+2
3

x
i+1

f (x) dx

xi

) f ( :
 a



a




m

E :=  f (x) dx P (x) dx O h3


b
b


o h5 h
. :
 a

n

 m
! 5

1
a
X






ba
n
m
5


= o h4
E :=  f (x) dx P (x) dx
o h = o
m
n


i=0
b

O h4 O h3 .

2.3
m E m O hm+2
m .E m O hm+1

2.4 m f .m + 2
f
.
.
29

2.2

2.2

:RN

:RN

[ai , bi ] RN

QN

i=1

= A : f : Rn R :
bN

f (x1 , ..., xN ) dxN dx1

aN

an

= f (x1 , ..., xN ) dA

a1

Newton-Cotes m



O hm+1 f m + 1
. n RN
N1 n

 m+1
  m+1
1

= O n N
.O :

N n
m N n .
n N m )
(.

. A RN )
( . .
2.2.1 :Monte-Carlo
A A" ) .U (A f : R R
1
] [a, b" )] U U ([a, b
p (x) = ba :
] f (x) dx = (b a) EU [f

=f (x) dx = I :

b
a

1
1
= dx
)f (x
ba
ba

= ] EU [f

I ] ,EU [f
i.i.d
)] X1 , ..., Xn U ([a, b :
n

ba X
f (Xi ) I
n i=1

=In :

In .I ,
i.i.d
D := [a, b] [c, d] R2 ) X1 , ..., Xn U (D :
n

(b a) (d c) X
f (Xi ) I
n
i=1

=In :

] [a, b] [c, d " ] [a, b ] [c, d :

n

(b a) (d c) X
f (Xi , Yi ) I
=In :
n
i=1
i.i.d
i.i.d
)] X1 , ..., Xn U ([a, b )] Y1 , ..., Yn U ([c, d .

30

2.2

:RN

QN
] A := i=1 [ai , bi f : RN R :
Qn

n
n
Y
(bi ai ) X
) f (X1,i , ..., XN,i
(bi ai ) EU[A] [f ] i=1
= f (x) dx
n
A
i=1
i=1
Q
i.i.d
)] Xj,i U ([ai , bi 1 i N 1 j n ) ni=1 (bi ai .
D RN N f : RN R :

n
1X
) f (Xi
)f (x) dx = V (D) EU[D] [f ] V (D
=I :
n i=1
D
)) Xi U (D D N (.
 
2.5 In I < EU f 2 In
,O 1n m ) N(.
,Monte-Carlo:


n
 V (D) X



=
 f (Xi ) I
 n

i=1

n
EMC

Xi RN .f : RN R n :
!
n
f2
V (D) X

f (Xi ) I N 0,
=Y :
n i=1
n

. ) X U (D:

1
1
= dx
f (x) dx
)f (x
= ])f = EX [f (X
)V (D
V (D) D
D
:
2

(f (x) f ) dx

1
1
= dx
)V (D
)V (D

) (f (x) f

n
E [EMC
]

#
"
n

 V (D) X


 f (Xi ) I
 E

 n

= ])f2 = Var [f (X

i=1

]| E [|Y
ds

s
Y

s2
22
Y

|s| e

= t:

1
ds = p
2Y2

)2

(sY
22
Y

Y N 0, Y2

|s| e

f2
n

= ,Y2 :

1
E [|Y |] = p
2Y2

=C

z
|}
{



2
2
t
f
Y
Y
2
|t| e dt = C O (Y ) = O
E [|Y |] = p
n
2
2Y2

31

:RN

2.2




n
E [EMC n ) N(
] O f 1n
.m f
. ) f (x) = C ( :
Cdx

V (D) X
= )f (X) = C V (D
1 i=1

f .f = 0
2.6 .

1
n


n
,E [EMC :f
] O f

2.2.2 :Importance Sampling

f


. .
p :
 
n
) 1 X f (Xi
)f (x
f

p (x) dx = Ep
)p (x
p
) n i=1 p (Xi

= f (x) dx

=I :

i.i.d
.X1 , ..., Xn p

1
n


n
.E [EIS
] O f
p

2.7 p fp C f = 0 p
p
I . p = C f f
.
p p p
,f p ) p (f . f < f
p

2.8 f (x) > 0 {f (xi )}i=1 Importance

Sampling p .f ) p (x
:
) 2 I(xxi

) |f (xi )| e(xxi

n
X

= )p (x

i=1

" |) |f (xi p
f f ,"
f < f .
p

2.2.3 :Inverse Transform Sampling

Inverse-Transform Sampling
F .p :

32

2.2

:RN

)] U U ([0, 1 F ) F 1" F,
F ( :




]F (x + h) F (x) = P (F (x) < U < F (x + h)) = P x F 1 (U ) x + h = P F 1 (U ) [x, x + h
) F 1 (U ,F .F 1 (U ) F
x
F ,p F (x) = p (s) ds :
x+h

p (s) ds

= )(U ) [x, x + h] = F (x + h) F (x

P F

F .p
: " ) F (:
)] U U ([0, 1 ) .F 1 (U F
F 1 F 1 .
:2 " ) p (:
x
:F F .F (x) = p (s) ds p
] Supp (p) [a, b
x
] [a, b ) x (a, b , F (x) = a p (s) ds x a F (x) = 0
x b .F (x) = 1
:F 1 F ] [0, 1 .
)] U U ([0, 1 .F 1 (U ) F
" ) F 1 (U .p
2.9 F 1
. "
.
F ) F F
( F } F 1 (u) = inf {x R | F (x) u
} F 1 (u) = sup {x R | F (x) u .
:Importance Sampling
i.i.d
f p ] [a, b X1 , ..., Xn p :
n

) 1 X f (Xi
) n i=1 p (Xi

f (x) dx

b
a

p Y p )] X U ([a, b

) f (Y
2
< Var (f (X)) = f2
f := Var
p
) p (Y




O f 1n O f 1n .


33

2.2

:RN

2.10 :
b
Pn
.1 ) a f (x) dx n1 i=1 f (Xi )] Xi U ([a, b
.2

Pn

) f (Xi
) i=1 p(Xi

1
n

f (x) dx

b
a

Xi p

1
n

1
n

p
.O
) Var (f

r
 
.O
Var fp

n . ) Var (f
n .
 
p ) Var fp Var (f n
.
2.2.4 :RN
.
p : RN R0
QN
) p (x1 , ..., xN ) = i=1 pi (xi pi , ..., pN .
2.11 :

N ) X N (0, IN " N"

i.i.d
) X1 , ..., XN N (0, 1 . N
1
Y = 2 X + ).N (,
2.12 ):(Mixture of Distributions
p1 , ..., pk : RN R0 :
) i pi (x1 , ..., xN

k
X

=P (x1 , ..., xN ) :

i=1

p1 , ..., pk )
pi (.

k = 1 .1 = 1 P }i {1, ..., k
i .pi " I } {1, ..., k
.P (I = i) = i i ) I ( i
.(X|I = i) pi " X } i {1, ..., k ,(X|I = i) pi
) ( :
)i pX|I (x1 , ..., xn |I = i

k
X
i=1

= )P (I = i) pX|I (x1 , ..., xn |I = i

) i pi (x1 , ..., xn ) = P (x1 , ..., xn

k
X

k
X

= ) pX (x1 , ..., xn

i=1

i=1

} i {1, ..., k (X|I = i) pi .X P

Inverse-Transform-Sampling
" I"

P
P
) q (x) = kj=1 i (x i .F (i) = ij=1 i
)] U U ([0, 1 i )) F (i) U F (i + 1" F F
( . i pi .

34

2.2

:RN

2.13 CDF
Inverse-Transform-Sampling 0
.
2.14 p~1 p~2 } .{1, ..., N
~ Rank (A) = 1 ,A = p~1 p
A i, j ) Aij = p1 (i) p2 (j 2
Aii ).p (i) = p1 (i) p2 (i

~
~p1 p
+
(1

)
q
~
~
q

~q1 , ~q2 } {1, ..., N

1 2
2
~q1 , ~q2 ~p1 , ~p2:


~Rank
~p1 p
q1 ~q2 = 2
~ )2 + (1

. } {1, ..., N N
} {1, ..., N } {1, ..., N Mixture
.
2.2.5 Rn :Rejection Sampling
D RN
D ) (
f : RN R D . , D A
D A f : RN R D :
n

   
V (A) X
~ 1 , ..., X
~ n i.i.d
~i ; X
~i f X
) U (A
1D X
n i=1

1D (~x) f (~x) d~x

= f (~x) d~x

A .
A\D
,D A\D
). V(D
. A D
)V(A

 p
1
) .O n Var (1D f
Var (1D f ) Var D f
 )
 (fp
1

.O
) Var (f
n
2.2.6

:
 .

.1 Newton-Cotes

)(m+1

O n N
) (m
) (N m.
 p

.2 Monte-Carlo ) ,O 1n Var (f
Newton-Cotes
.

35

 3 :

f (x) = 0 f : R R
. .
3.1 x R3 kx xi k = di i = 1, 2, 3
.x ) GPS
(.

3.1

):(Bisection Method

f : R R x R ,f (x) = 0 a, b R
) f (a) f (b) < 0 ) f (a ) f (b (
f ] c [a, b .f (c) = 0 :
) f (c

a+b
2

= c a b c ] [a, c ].[c, b

|a b| > |f (c)| .c
f (b) > 0 f (a) < 0 :
BM(f, a, b, , ):
while |a b| > :
c = a+b
2
if f (c) > 0
b=c
else
a=c
if |f (c)| < exit
end.
en .n |e0 = |b a
:
 
 


 
 b a  en
a + b
 a + b



=

a ,  b
=
 en+1 = max {|c a| , |b c|} = max
2
 2
 2
2
< 1

1
2

en+1
en





lim  en+1 ) . (0,
3.2 ) , (...,  <
e
n
n

:
e0 n
= ... = n 0
2

3.2

en1
en
=
2
4

= en+1

:Newton-Raphson

3.2.1 :
f : R R ) (
x . x1
x0 R f (x) = 0
:

f (x0 ) (x x0 ) + f (x0 ) = 0
36

3.2

3 :

:Newton-Raphson

" f x0 . x1
x2 f x1 .
) x " " (
x0

{xn }n=1 xi :

f (xi1 ) (x xi1 ) + f (xi1 ) = 0

) i1
"
. :
) xi = xi1 ff(x " ( x
(x
)
i1

) f (xi
) f (xi

Newton(f, x0 , ):
while |f (xi )| > :
set xi+1 = xi

,n ,en = xn x :

) f (xn
) en f (xn ) f (xn
) f (xn
x
= en
=
= xn+1 x
= xn
) f (xn
) f (xn
) f (xn

en+1

[ (n :
) n [xn , x ] x, xn
x ]
f
en

z }| { 1
2
( 0 = f
( ) x) = f (xn ) + f (xn
) x xn ) f (n
( x xn ) +
2

1
1
2
(
) x xn ) f (n ) = e2n f (n
2
2

= ) en f (xn ) f (xn

:




 )  en+1  1  f (n
) 1 2 f (n




=  2  = 
= en
) 2 f (xn
en
 ) 2 f (xn

en+1



 (n )  n
. ff (x :
"  < 0
)n
Mn

z
|}
{


 ) f (x
 )  f (n
 |en | |en | sup 

|en+1 | = |en | en


 ) 2f (xn
)x,yIn 2f (y

.rn |xn x :
[ = In |
x rn , x
] + rn
Mn

n
Y

i=1

| |en+1 | = |en | Mn = |en1 | Mn1 Mn = ... = |e0

Mn C < 1 n :
n

|en+1 | < |e0 | C n 0 = xn x

x :
x0


 ) f (x
 = M0 < 1
|e0 | sup 

)x,yI0 2f (y

.
| = .r xn x
[ = I0 | x x0
x r, x
] + r
37

 4 :

3.3 f (x) 6= 0 < ) f (x .

3.4 f (x) 6= 0 .
x
x ) f (x
x0 I0 .

( f :
( x) = f
x x) = 0
f (x) 6= 0


 ) f (x
<=c

lim 
x
 )x f (x

.
.

3.5 .f
3.6 f , |f (x) = 1 |x
x = 0 x0 .x0 = 0

| 1 |x ) f (xn
.
3.2.2 Newton-Raphson :RN
f : RN RN ) (fi
~x0 RN .f (~x) = ~0 n
f g n : RN RN 1 i N:
gin (~x) = fi (~xn ) + h(~x ~xn ) , fi (~xn )i
~xn+1 .g n (~xn+1 ) = ~0
:
) ~0 = g n (~xn+1 ) = f (~xn ) + Jf (~xn ) (~xn+1 ~xn
) Jf (~xn f ) ~xn
:

~( fi
) xn
xj

= ([Jf (~xn )]ij ) Jn (~xn

) ~xn+1 = ~xn Jf1 (~xn ) f (~xn

) Jf (~xn
) f (~xn ) = Jf (~xn ) (~xn+1 ~xn
.
3.7 .f

N N Ax = b .A RN N
A .x = A1 b
:
: .
38

 4.1 LU :

4 :

: .
4.1 A ' )Gauss-Jordan
Reduced Row-Echelon
(Elimination ]) [A|b


.(Form " .O N 3

4.1 LU :

A RN N L, U RN N L U , .A = LU
Ax = b A = LU LU x = b z = U x .Lz = b L
Lz = b Forward-Substitution :

X
1
b1
lij zj
= , for i = 2, ..., N : zi
bi
= z1
l11
lii
j<i
U x = z Back-Substition U :

X
1
zN
uij xj
= , for i = N 1 : xi
zi
= xN
uN N
uii
j>i

Forward-Substition Back-Substition LU .O N 2

4.2 LU '

Ax = b b
. '



] [A|b O N 3 LU O N 3


O N 2 Forward-Substition .Back-Substition
4.1.1

LU " :

4.3 LU
. .
A RN N ,LU L U
:
!
X
X
1
= i j = aij
= lik ukj = uij
aij
lik ukj
lii
ki
k<i

X
X
1
aij
lik ukj
= i j = aij
= lik ukj = lij
ujj
k<j

kj

A uii lii :
lii = uii , lii 6= 0

N
Y

i=1

uii

N
Y

i=1

= )0 6= det (A) = det (U ) det (L

LU = A " N 2" N 2 + N
) L U ( .
N L, U N 2 N 2 .
a1j
u1j = l11
li1 = ai1 1 i N uii = 1 1 i N
39

: LU 4.1

: 4

U L . 2 j N
:Crouts Algorithm
LUcrout(A):
for i = 1, ..., N :
li1 = ai1 and uii = 1.
end.
for j = 2, ..., N :
u1j =
end

a1j
l11

a1j
a11 .

for i = 2, ..., N :
for j = 2, ..., i:
lij =
end

aij

lik ukj

k<j

for j = i + 1, ...., N :


P
1
lik ukj
uij = lii aij
k<i

end

end

40

 4.1 LU :

4 :

uii = 1 i :
LUdecomp(A):
for i = 1, ..., N :
for j = i, ..., N :


P
ljk uki
lji = aji
k<i

end
for j = i + 1, ..., N :


P
1
uij = lii aij
lik ukj
k<i

end
end

O N 3 .

4.4 A uii , lii 6= 0 i l11 = a11

a11 = 0 . A a11 = 0
) ( . LU
.a11 = 0 LU P
P P A = LU .
4.5 LU A RN N .
L U .
".
4.6 LU ,

1 2
3
A = 2 5 12
0 2 10

a21 .
2 a22:

1 2
3
6
A(1) = (2) 1
0
2 10

3
6
2

,a32 :

1
2
A(2) = (2) 1
)0 (2

) ( A:

1 2 3
U = 0 1 6
0 0 2

0 0
1 0
2 1

1
L = 2
0

LU = A L U
) ( .A
41

 4.1 LU :

4.1.2

4 :

LU :A

4.7 :A RN N
AIJ A } I {1, ..., N
} .J {1, ..., N
} .I = J {1, ..., N
} I = J = {k, ..., N 1 k N
N N N .
4.8
A LU" A.
4.9 rankA = k LU k .
LU .
: , A . M RN N
M k } I = J = {k, ..., N .1 k N
k LU :A
: k = 1 AN , k = 2 Ak
u, l .u l = Ak

: LU Ak .Ak+1 Lk , Uk
,Ak = Lk Uk Ak Lk , U k .
u Rk :
Lk u = A1k,k+1
A1k,k+1 k k + 1 .A Lk u :
A1k,k+1

1

u = Lk

l Rk k) Ak+1,1k k + 1 (A:
Ak+1,1k
:

1

l = Uk

u l + uk+1 vk+1 = Ak+1,k+1

uk+1 lk+1 , ) (l, lk+1
) (u, uk+1 U k , Lk :




] [Lk
0
] [Uk
u
= , Lk+1
= Uk+1
l
lk+1
0
uk+1
Lk+1 Uk+1 = Ak+1 LU .Ak+1

42

 4.2 Cholesky :

4 :

4.2

Cholesky :

4.10 Cholesky :
) Cholesky (L L A RN N L RN N A = L L L
) L (.
4.11
A L L" .
= : L L A :
= L L = A

A = L L = A = L L

L L :
x Ax = x L Lx = kLxk 0
A L kLxk > 0 x RN x Ax > 0 .x RN
=
. A = U U U \ " A
1
1
" A . ) L = 2 U 2 "
( .A = L L L" , Q
) (QQ = I = Q Q QL :

A = L L = L IL = L Q Q L = (QL) QL
= RL
L . U
1
2 U :
  1   1
1
1
2 U 2 U
= 2 U U 2 =

A :

  1   1
1
1
2 U
2 U = U 2 2 U = A

L = 2 U v1 , ..., vN } .{vN , ..., v1

r1 , ..., rN , ri rj i 6= j
ri vj ) j > N i + 1 ( hri , vj i = 0 j > N i + 1 :

hr , v i hr , v i hr , v i
0
N
1
N
N
N
1
rN

..
..
..
..
..
..

v1 vN =

=
:= L
.
.
.
.
.
.
r1
hr1 , v1 i hr1 , vN i
hr1 , v1 i hr1 , vN i
|
{z
| }
{z
}
L

L
L
= RL
.A = L

4.12 L L L .

43

4.3

4 :

4.2.1 Cholesky :
A = L L :
v
u
k1
k1
u
X
X
2
2
t
2
lk,i = lkk = akk
= lkk = akk
lki

2
lk,i

i=1

i=1

k
X

= akk

i=1

k L "
. j = k + 1, ..., N :
lki lji

k1
X
i=1

lki lji = lkk ljk = akj

!

lki lji

k1
X
i=1

akj

k
X
i=1

1
lkk

Lki L
= ij

N
X

= akj

i=1

= = ljk

Cholesky(A):
for k = 1, ..., N :
q
Pk1 2
lkk = akk i=1
lki

for j = k + 1, ...., N :


Pk1
1
akj i=1 lki lji
ljk = lkk
end

end


O N 3 .

4.13 L L LU LU
L L.
4.14 L L
) (.

4.3

."
)( .
Ax = b , Q :
Ax = b = (Q Q) x + Ax = b = Qx = b (A Q) x
)= x = Q1 (b (A Q) x

) x = Q1 (b (A Q) x :
) xn+1 = Q1 (b (A Q) xn

x0 . A1
Q1" " ,A
. Q:
:Gauss-Siedel Q .A
:Jacobi Q .A
:Richardson Q = cI c 6= 0.
44

4.3

4 :

4.3.1 :
4.15 < 1 p RN :
! p1

| |xi

N
X

= kxkp

i=1

" " | .kxk = maxi |xi

4.16 :
A RN N kkp kkq A :
kAxkq
kxkp

sup
06=xRN

= kAkp,q

" RN N .1 p, q
4.17 p = q .kAkp
4.18 0 6= x RN p, q :
kxkp kAkp,q

kAxkq
kxkp

kAxkq = kxkp

4.19 RN N RN
, .
4.20 p, q :
.1 p = q = 2 kM k2 = maxi i 1 , ..., N Singular Values .M
PN
kM k1 = max .
.2 p, q = 1 | i=1 |mij
1jN

.3 = p, q | |mij

PN

j=1

kM k = max .

1iN

4.3.2 :
) xn+1 = Q1 (b (A Q) xn .
A :
.en = xn x x = b
" x n
en+1 = xn+1 x
= Q1 (b (A Q) xn ) x

= Q1 (A
x (A Q) xn ) x
( = Q1 (A
x xn ) + Qxn ) x

n
1
n
n
n
1
n
1
( = Q A
xx )+x x
= e Q Ae = I Q A e

n+1
Operator Norm
e

{|}z




p
n
1

I Q1 A p

= I Q A p ke kp
n
ke kp
45

n+1


e
= I Q1 Aen
p
p

4.3

4 :

) I Q1 A p < 1 p( :



n+1




e
I Q1 A ken k I Q1 A 2 en1 .... I Q1 A n e0 n
0
p
p
p
p
p
p
p

, .

4.21 Q = A I Q1 A p = 0 .
4.22 :
M RN N " 1 , ..., N A" | . (A) = maxi |i





A RN N I Q1 A V = v 1 , ..., v N
. en RN en
1 , ..., N"PN
V" .en = i=1 eni v i en V :
N
N
X



n

X
n i
1
eni I Q1 A v i
= ei v
A e = I Q A
i=1

e0i ni v i

N
X

i=1

= ein1 2i v i = ...

i=1

N
X

= eni i v i = ...

i=1

= I Q

N
X

n+1

i=1

p:
n
))=((IQ1 A
=1
Orthonormality N
N
|}
{
z
N

X

n X
X
{ |}  z
n+1
 0


{|}z
n 0 i
| n e0  v i max |i
e
=
e  = I Q1 A n e0
=

v


i
i
i
i
i
p
1
p
i

i=1

i=1

i=1

n
I Q1 A < 1 ken kp 0 . :
def
N
X
{|}z
eni i v i
en+1
= v i = en+1
i
i=1

N
X
i=1

} {v1 , ..., vN en+1 "

en+1 1 i N :
= i eni
i
en+1 = Diag (1 , ..., N ) en = Diag (n1 , ..., nN ) e0

 
 n


 en+1  = |i |n e0 maxi |i | < 1  0
en+1 .1 i N
i
i
i

4.23 Q, A I Q1 A

. 1 , ..., N I Q1 A = maxi i
2
maxi i < 1 .


4.24 I Q1 A > 1 i ) i > 1" (I Q1 A:
 
  n+1
e  = |i |n e0i  n

i

. I Q1 A < 1
) .xn+1 = Q1 (b (A Q) xn
46

4.3

4 :

4.3.3 :Jacobi
:Jacobi
Jacobi ) Q = Diag (A

1
1
a11 , ..., aN N

i=j
i 6= j
:

0
aij
aii

= Diag

= A ij

,Q A

aij
aii

I Q

 N
 X  aij
X
X





1
  = max 1
I Q1 A = max
=
max
|aij |
I

Q
A




ij

1iN
1iN
1iN
aii
| |aii
j=1
j6=i

j6=i

1

x0 Jacobi
P I Q A < 1
. 1 i N
A | j6=i |aij | < |aii
, .Diagonally Dominant Matrices

47

4.3

4 :

:Jacobi
> 0 :
JacobiSolver(A, x0 ):
while krn k2 = kAxn bk2 :
for i = 1, ..., N :


P
1
n
b

A
x
xn+1
=
i
i
j6=i ij j
aii
end
end

P
A ) A 0 ) (O (N j6=i Aij xnj


) O (1
) ,O (N A .O N 2
O N 3 A .

N .
4.3.4

:Gauss-Seidel

:
A RN N A = Q + U Q A .U = A Q v"


I Q1 A " " ) kvk = 1 " v "
,( kvkv :

1
U v = (Q A) v = Qv Av = Q I Q A v = Qv
U v = Qv 1 i N:
aij vj + aii vi

i1
X

aij vj =

j=1

aij vj

N
X

j=i+1

aij vj +

i
X

aij vj =

j=1

i1
X
j=1

N
X

j=i+1

= aii vi =




 i1
N
i1
N
X
X
X

 X

| |aij vj
|aij vj | +
|aij vj  |
aij vj +
|aii vi | 

 j=1
j=i+1
j=1
j=i+1

i1
X
X
| kvk |
) |aij | = (|| S1 + S2
|aij | +
} | {z
j=i+1
j=1
=1

kvk = 1 i |vi | = 1 i" :

|| |aii | = (|| S1 + S2 ) = || (|aii | S1 ) = S2

A Diagonally Dominant :
|aij | = S1 + S2 = |aii | S2 > S1

X
j6=i

> | |aii

= S2 = || (|aii | S1 ) > || S2 = || < 1

A Diagonally Dominant " I Q1 A 1 .
48

4.3

4 :

4.25 A .
:Gauss-Seidel
Q A :
xn+1 = Q1 (b (A Q) xn ) = Qxn+1 = b (A Q) xn

= Diag (A) xn+1 = b (A Q) xn + (Q Diag (A)) xn+1

1 i N:
aij xn+1
j

i1
X
j=1

i1
X

aij xnj +

N
X

j=i+1
N
X

aii xn+1
= bi
i

aij xn+1
aij xnj +
bi
j
aii
j=1
j=i+1

= xn+1
=
i

Diagonal-Dominance aii = 0
A " .
" :
GS-Solver(A, x0 ):
while krn k2 = kAxn bk2 :
for i = 1, ..., N :


P
Pi1
n+1
n
xn+1
= a1ii bi N
i
j=i+1 aij xj +
j=1 aij xj
end
end

xn In-Place ) O (N A O N 2 A.

4.26 " GS .Jacobi

4.3.5 :Richardson

)
.(kAxn bk2 < Richardson Q = cI c 6= 0
. :
Qxn+1 = b (A Q) xn = cxn+1 = b (A cI) xn
1
) = xn+1 = xn + (b Axn
c
:




1
en+1 = I Q1 A en = I A en
c


1 , ..., N " ,A " I 1c A 1 1c
i "
.i , I 1c A < 1
c 6= 0 . A i > 0 i :





1
i
 i
i

I A = max 1  < 1 i : 1 < 1
< < 1 i : 0
<2
i
c
c
c
c
max
1
2
max
> c
< < < 2 0
< 0
c
c
max
2
49

4.3

4 :

max " .A A
c > max
2
.Richardson
c > max .
2
} V = {v1 , ..., vN I 1c A " A .


en+1 i " . I Q1 A
en V = i eni
i
Q = cI 1 i N:



n
 

 n

  
|   = 1 i
 1 i  e0
: en+1
e
|
=
...
=
i
i
i



c
c


1
A
< 1
" "(

c


 i

c . 1 c

c > max :
c
2



min
 i
max 1  = 1
i
c
c
.max

 min


 i
c = max
c > max mini 1 max  = 0
2

min
max

.1





. (A) =  max
4.27 A RN N " 1 , ..., N Condition Number A" 
min
V en = en V ,A :
v



n
n
n u
N
uX











2
1 min  e0 = max 1 i  e0 = max 1 i  t
|
| e0i

2
2
 i
 i
 c
c
c
i=1

v
v
 uN
 uN


n 2
n
2
uX
uX




i  0
2
i
t
0

 max 1
|  max 1
=
| ei
|
ei | = t
i
i
c
c
i=1
i=1
v
v
 u N
n
2 u
N
 uX
X
n+1
 n+1 2 n+1 2 n+1 2 V is orthonormal


{|}z
z}|{ u
i
t
t
0

e

e  = e = V e
|  1
=
|
e
=

i
i


2
2
2
c
i=1
i=1

<

min n
c


1 en+1 2 < c = max

 n
n




1 1  = 1 min  <



) (A
 max

a) n = (A) a( ) := (A
 n
a

1
n
1
= 1
ea = e
1

:
n

n
1
n
) e < = < log = n = ( log
1

0 < 1 log . ) (A
ken
k2 < )| O ( (A) |log Richardson
)) O (N (A A ) O N 2 (A A.
50

4.3

4 :

4.28 GS .Jacobi
4.29

)) O (N (A A
) O N 2 (A A ) (A) O (N .
b = A :
n .rn := b Axn x


x xn ) = Aen = kAxn bk2 = krn k2 = kAen k2 = ken k2 = A1 rn 2
}rn = |{z
A
( x Axn = A
b

en ken k2 <
.krn k2


ken k2 = A1 rn 2
krn k2 <
.A
A1 rn 2 = ken k2 A1 . ken k2 krn k2
) , (A ) (A .
Pre-Conditioning Condition-Number:
) (A
Condition Number . Ax = b , (A) 1 P
.P Ax = P b ) (P A) (A
.P Ax = P b P .left-preconditioning right-preconditioning
AP y = b x = P y ) . (AP ) (A pre-conditioning
P AQy = P b )) (P AQ) (A Q = P (.
4.30 Diagonally-
Dominant .
4.3.6 :Steepest-Descent

A f : RN R f (x) = x 2Ax x b
f (x) = Ax b f .Ax = b
Ax = b ) .minxRN f (x ":
4.31 B RN N RN RN hu, viB = u Bv B
) " B ( "
2

kvkB = hu, uiB = u Bu

A ) ( A1 :


) kAx bkA1 = x Ax 2x b = 2f (x

) f (x kAx bkA1 .

: f
.dn
dn RN ) minn f (xn + n dn n . :

]f (xn + n dn ) = (dn ) f (xn + n dn ) = (dn ) [A (xn + n dn ) b

n
= (dn ) (Axn b) + n (dn ) Adn
51

4.3

4 :

(A ":
) en = xn x x = b
"
rn = b Axn = (A
( x Axn ) = A
x xn ) = Aen
:

f (xn + n dn ) = (dn ) rn + n (dn ) Adn

n
) minn f (xn + n dn :

f (xn + n dn ) = (dn ) rn + n (dn ) Adn

n

(dn ) rn

= = n (dn ) Adn = (dn ) rn = n

(dn ) Adn

=0

A (dn ) Adn > 0 " .

Steepest-Descent :
0
Steepest-Descent(A, x ):
while kAxn bk2 > :
set dn = b Axn = rn
(dn ) r n
(dn ) Adn
n+1

= xn + n rn

= n

update x
end

4.32 dn = b Axn = rn
) f (x ) (.


: ) O (N O N 2
. " ) (
))) O ( (A ) (A
( ." )) O (N (A

A ) O N 2 (A A.
:
xn+1 x
= xn x + n rn = en+1 = en+1 + n rn
:
.1 A " ) v RN "( , :


0
2 e
e0
A Ae0
1

=
=

0
3
0

e A e

0 0
e
3 e

e0

r0

r0

(r0 ) Ar0

:
1 0
e = 0

e1 = e0 + 0 r0 = e0 0 Ae0 = e0 0 e0 = e0

52

4.3

4 :

.2 e0 " A " , :
e1 = e0 + 0 r0 = e0 0 Ae0 = e0 0 e0
0 = 1 e1 = 0 .
" A .
: :
!

(dn ) rn n
n+1
n
n n
n
= b Ax
= b A (x + d ) = b A x +
r

(dn ) Adn
!

(dn ) rn
(dn ) rn
n
n
= b Ax
Ar = I
A rn

| {z } (dn ) Adn
(dn ) Adn

rn+1

rn

A rn

(dn ) rn

(dn ) Adn

) dn , rn+1 = (dn ) rn+1 = (dn

(dn ) rn n

(d) Arn = (dn ) rn (dn ) rn = 0

n
n

(d ) Ar

= (dn ) rn


, dn , rn+1 = 0 . :

n n+1
= (dn ) Aen+1 = (dn ) Aen+1 = (rn ) rn+1 = 0
d ,e
A

h, iA .



4.33 dn = rn , dn , dn+1 = rn , rn+1 = 0 .
 N 1
: di i=0 A RN N
eN +1 h, iA N .eN +1 = 0 ,:

en+1 = xn+1 x
= xn x
+ n dn = en + n dn
 i N 1
P 1
i
0
e0 = N:
e d i=0 i=0 ai d
ai di + 0 d0

N
1
X

0 0

= e =e + d

i=0

 i N
(d0 ) r0
0 = (d0 ) Ad0 e1 A d0 d0 e1 d i=1
PN 1
.0 = a0 e1 = i=1 ai di

(dn ) r n
(dn ) Adn

= n n < N :

ai di + n dn = n = an

N
1
X

= en+1 = en + n dn

i=n

 N 1
di i=0 A :
aN 1

{ |} z
+ N 1 dN 1 = 0

N 1

= aN 1 d

N 1 N 1

ai d +

N
1
X

i=N 1

53

N 1 N 1

N 1

=e

4.3

4 :

A:
Improved-SD(A, x0 ):
while krn k2 = kAxn bk2 > :


Pn1
dn = rn i=0 di di , rn A
  n n
r
dn
xn+1 = xn + (d(dn )) Ad
n
end

: " N
.

.


O N 2 A O N 3

p
A . ) (A
O
 p

 p

) O N 2 (A A ) O N 3 (A A.
4.3.7 :Conjugate-Gradient

 p

 p
) O N (A A ) O N 2 (A A.

SD d0 , ..., dN 1 A
en+1 A di .1 i n d0 = r0 :




Dn = Span d0 , ..., dn = Span r0 , ..., rn
r0 , ..., rn " .d0 , ..., dn :

rn+1 = b Axn+1 = b A (xn + n dn ) = b Axn n Adn = rn n Adn

:




Dn = Span r0 , ..., rn = Span r0 , r0 + 0 Ad0 , r1 + 1 Ad1 , ..., rn1 + n1 Adn1



= Span d0 , Ad0 , Ad1 , ..., Adn1 = Span d0 + ADn1 = Dn1 + ADn1

rn+1 di" en+1 A di " 1 i n

rn+1 Dn = Dn1 + ADn1 = rn+1 ADn1 = rn+1 A Dn1


di , rn A = 0 1 i n 2:


di di , rn A = rn + n1 dn1 , rn A

n1
X

dn = rn +

i=0

Improved-SD Conjugate-Gradient :
Conjate-Gradient(A, x0 ):
while krn k2 = kAxn bk2 > :


dn = rn + n1 dn1 , rn A
  n n
r
xn+1 = xn + (d(dn )) Ad
dn
n
end.

54

):(Eigen-Decomposition

) O (N A O N 2
A . .
:
Sparse


 )O N 1 (A
p
)O N 2 (A
 p

)O N (A

Dense


 )O N 2 (A
p
)O N 3 (A
 p

)O N 2 (A

Jacobi, GS, Richardson, SD

Improved-SD
Conjugate-Gradient

4.34 Steepest-Descent Conjugate-Gradient Positive-

) SemiDefinite Positive-Definite .(SemiDefinite
GS/Jacobi/Richardson .

5 ):(Eigen-Decomposition
A RN N )( , " " ".

5.1

:Power-Method

} V = {v1 , ..., vN A RN N " .N < ... < 2 < 1

x0 RN :
Axn
An x0
=
n
kAx k2
kAn x0 k2
x0 V ai vi

PN

i=1

= xn+1

= x0 :

PN
PN
PN
n
An1 i=1 ai vi
An1 i=1 ai Avi
An x0
i=1 ai vi i





=
=
...
=
=
=

n1 PN
n1 PN
PN


kAn x0 k2
A
A
i=1 ai vi ni
i=1 ai Avi
i=1 ai vi

n+1

PN
PN
n1
n
z}|{ a1 n1
ai ni
{|}z
i=1 ai vi i
v1 = sgn (a1 n1 ) v1
= qP
= qP i=1
vi
| |a1 n1
2
2
N
N
n
n
) i=1 (ai i
) i=1 (ai i

} {v1 , ..., vN:

*v
+ v
u N
uN N
N
X
u X
uX X
n
n
t
=
ai n aj n hvi vj i
aj vj = t
ai vi ,
j

i=1 j=1

j=1

i=1

i=1

v
v
uN N
u n
uX X
uX
n
n
t
ai i aj j ij = t
(ai ni )2
i=1

N

X

n
ai i vi

i=1 j=1

) xn v1 ( , , ) (ai ni
N
X
ai ni
a1 n1
v1 +
vi
D
D
i=2
|
{z
}

= vi

Error

55

N
X
ai n
i

i=1

= xn+1

qP
N

i=1

= D ::

5.1

5 ):(Eigen-Decomposition

:Power-Method

N < ... < 2 < 1 :

qP
v
Orthonormality u

   n
N
N
n 2
N

X
n
n
u
X
  2   a2
ai i
1t
{|}z
| i=2 (ai i ) z}|{ N |2 | |a2


2
n
q
q
(ai i ) = P
    = N
=
vi
<
= kEk2


D
D i=2
1
a1
N
2
n 2
i=2
) (a1 n1
2
) i=1 (ai i
n | |n2 a2 | |ni ai .2 i N :
 
 
 n
 
  
) log log  aa21  + log (N
  2
  a2
|  2  |a2
 
> < = log   N + n log   < log = n
  kEk2 < N
 
1
| |a1
a1
1
 log  21

,, N a1 , a2
x0 2 , 1 n 1 2 .

 
 
5.1   12 Spectral-Gap .v1


  i
 i+1 i .

5.2 a1 = 0 " "
.x0
2 6= 1 ,v1 vk+1 .v1 , ..., vk

,:
.1 x0 xn+1 = Axn .
.2 "


Pk
xn+1 i=1 xn+1 , vk vk
n+1


x
=
Pk

xn+1 i=1 hxn+1 , vk i vk

.3 ) xn (.
)

5.3 
Pk
x0 = x0 i=1 xn+1 , vk vk xn+1 = Axn( ,
) (.
} {v1 , ..., vN :v Av
kvk2 =1

{|}z
i = i v v = v v = v Av
:

 

 a
) log log  a2  +log(N
  1
 
 log  2
1

O :

A O N 2 : " O N 2 + N k 2 k .




A O (N ) : " O N + N k 2 = O N k 2 k .
56

 :Inverse Power-Method

5.2

5 ):(Eigen-Decomposition

5.4 " : " "

. ,
" " Eigenspace
" .
" "
" . " "
," " . .

5.2

:Inverse Power-Method

A RN N :
.1 .(A I) xn+1 = xn
.2

xn+1
kxn+1 k

= .xn+1

" (A I) xn+1 = xn
1
.xn+1 = (A I) xn ) A
( ) N
( i " A " ) (A I
i = i1 :
| argmini |i | = argmaxi |i
) i"
" ( .
i
i+1 Spectral-Gap Power-Method
i = i+1
) ,(A I" .Inverse Power-Methods

: "
) Jacobi/GS( " \ .
Power-Method k
.k 2

5.3 ) Lanczos (:
A Ai x0 .1 i n Power-Method
An x0 .v1 Lanczos
. uk = Ak x0 ,0 i N 1
N 1
{   U .N N
ui }i=0

ui , u
,TA := U AU j iA
U AU ij = h TA
) (Conjugate-Gradient A TA
) [TA ]ij = 0 .(|i j| > 1
5.5 A hu, viA
) hu, uiA 0 A (.
U :
TA = U AU = U 1 AU

57

5.4

) Jacobi Iteration (:

5 ):(Eigen-Decomposition

A UA
" . TA
" " ) " QR (O N 2 . 1 , ..., N " TA
) (A " v1 , ..., vN vi = U vi " A TA = U AU = A = U TA U:
A
vi = U TA U vi = U TA U U vi = U TA vi = U i vi = i U vi = i vi





N
: Ai x0 i=0 ,O N 3 O N 3 " " TA






.O N 2 " O N 3 " " ) A O N 4 .(Power-Method

5.6 Arnoldi Lanczos .

5.4 ) Jacobi Iteration (:

A RN N .A = U U
1 i, j N ] (0, 2 Ri,j, :

cos
k, l = i, i

sin

k,
l = i, j

sin k, l = j, i
= [Ri,j, ]kl

cos
k, l = j, j

1
other diagonal elements

0
otherwise

,A = Ri,j, A Ri,j, :

) Akl = cos2 sin2 Akl + sin cos (Akk All

k 6= l Akl 6= 0 ) Akl = 0 ( :


cos2 sin2
1
Akk All
Akk All
1
= )2 cot (2
= = cot
=
sin cos
Akl
2
2Akl

" .Akl = 0
A .
, Ri,j, i 6= j
2

A2kl = kAkF O

< Akl2

X
k6=l

i6=j

2

A

FO

qP
2
= ) kBkF O (
k6=l Bkl

. :
Jacobi-Iteration(A):
set R(0) = I , A(0) = A


While A(n) F O > :
)(n

find k 6= l such that Akl 6= 0:

 ) (n) (n
Akk All
1
1
compute = 2 cot
)(n
2Akl

58

5.4

) Jacobi Iteration (:

5 ):(Eigen-Decomposition

)update R(n+1) = Rk,l, R(n

update A(n+1) = Rk,l, A(n) Rk,l,

end
) A(n . )A(n
) A(n "
.A ) R(n
U " .A O N 3
).O (log

59