13 views

Uploaded by Cédric Mzt

- Unstable&VeryUnstable
- SOLUTION OF EQUATION AX + XB = C BY INVERSION OF AN M ×M OR N × N MATRIX
- Linear Algebra MATLAB
- Matrix Revise
- Saad Y., Iterative Methods for Sparse Linear Systems
- Operator equation
- A. Huseynov, G.S. Guseinov
- Matrices
- A Rank-1 Update AlgorithFastSolvKernel Foleyamp; SammonDiscriminant Vect-Dkz
- Zakpoane Paper
- Matlab Intro
- Study Notes on Matrix Algebra PrepLadder
- 1-9
- Matrix Norm
- HW12
- Math for Machine Learning
- Non Singular
- Twodof Base Disp Modal RevA
- Math2081 2010C Midterm Sol
- FDLF

You are on page 1of 16

Version 1.0

John Stensby

The analysis of matrix-based algorithms often requires use of matrix norms.

These

algorithms need a way to quantify the "size" of a matrix or the "distance" between two matrices.

For example, suppose an algorithm only works well with full-rank, nn matrices, and it produces

inaccurate results when supplied with a nearly rank deficit matrix. Obviously, the concept of rank (also known as numerical rank), defined by

rank ( A , ) = min rank ( B)

AB

(4-1)

is of interest here. All matrices B that are "within" of A are examined when computing the rank of A.

We define a matrix norm in terms of a given vector norm; in our work, we use only the pr

vector norm, denoted as X p . Let A be an mn matrix, and define

r

AX p

,

A p = sup

r

r

Xp

X 0

(4-2)

where "sup" stands for supremum, also known as least upper bound. Note that we use the same

p notation for both vector and matrix norms. However, the meaning should be clear from

context.

Since the matrix norm is defined in terms of the vector norm, we say that the matrix norm

is subordinate to the vector norm. Also, we say that the matrix norm is induced by the vector

norm.

r

Now, since X p is a scalar, we have

r

r r

AX p

/ Xp .

A

X

A p = sup

=

sup

r

r

r

p

X

X 0

X 0

p

CH4.DOC

(4-3)

Page 4-1

EE448/528

Version 1.0

John Stensby

r r

In (4-3), note that X / X p has unit length; for this reason, we can express the norm of A in terms

of a supremum over all vectors of unit length,

A p = rsup

X p =1

r

AX p .

(4-4)

r

r

That is, A p is the supremum of AX p on the unit ball X p = 1.

Careful consideration of (4-2) reveals that

r

r

AX p A p X p

(4-5)

r

r

r

r

for all X. However, AX p is a continuous function of X, and the unit ball X p = 1 is closed and

bounded (real analysis books, it is said to be compact). Now, on a closed and bounded set, a

continuous function always achieves its maximum and minimum values. Hence, in the definition

of the matrix norm, we can replace the "sup" with "max" and write

r

AX p

r

= max

A p = max

r

r

r

AX p .

X 0

Xp

X p =1

(4-6)

When computing the norm of A, the definition is used as a starting point. The process has

two steps.

r

r

r

1) Find a "candidate" for the norm, call it K for now, that satisfies AX p K X p for all X.

r

r

r

2) Find at least one nonzero X0 for which AX 0 p = K X 0 p

Then, you have your norm: set A p = K.

From an application standpoint, the 1-norm, 2-norm and the -norm are amoung the most

CH4.DOC

Page 4-2

EE448/528

Version 1.0

John Stensby

important; MatLab computes these matrix norms. In MatLab, the 1-norm, 2-norm and -norm

are invoked by the statements norm(A,1), norm(A,2), and norm(A,inf), respectively.

The 2-norm is the default in MatLab. The statement norm(A) is interpreted as norm(A,2) by

MatLab. Since the 2-norm used in the majority of applications, we will adopt it as our default. In

what follows, an "un-designated" norm A is to be intrepreted as the 2-norm A 2 .

Recall that the vector 1-norm is given by

n

r

X 1 = xi .

(4-7)

i =1

A 1 = max

j

F a I.

GH i ij JK

(4-8)

That is, the matrix 1-norm is the maximum of the column sums. To see this, let mn matrix A

be represented in the column format

r

r

r

A = A1 A 2 L A n .

(4-9)

n r

r

r

r

r r

AX = A1 A 2 L A n X = A k x k ,

(4-10)

k =1

CH4.DOC

Page 4-3

EE448/528

Version 1.0

John Stensby

where xk, 1 k n, are the components of arbitrary vector X. The triangle inequality and

standard analysis applied to the norm of (4-10) yields

n r

x

r

r

AX 1 = A k x k x k A k 1

k =1

r

max A j

j

k =1

Fnx I

1GH k JK

k =1

(4-11)

r

r

= max A j

X

1.

1

j

With the development of (4-11), we have completed step #1 for computing the matrix 1-norm.

That is, we have found a constant

r

K = max A j

1

j

r

r

r

such that AX 1 K X 1 for all X. Step 2 requires us to find at least one vector for which we

r

r

have equality in (4-11). But this is easy; to maximize AX 1 , it is natural to select the X0 that puts

"all of the allowable weight" on the component that will "pull-out" the maximum column sum.

That is, the "optimum" vector is

r

X0 = 0 0 L 0 1 0 L 0

jth-position

where j is the index that satisfies

r

r

A j = max A k

1

1

k

CH4.DOC

Page 4-4

EE448/528

Version 1.0

John Stensby

(the sum of the magnitudes in the jth column is equal to, or larger than, the sum of the magnitudes

in any column). When X0 is used, we have equality in (4-11), and we have completed step #2, so

(4-8) is the matrix 1-norm.

Recall that the vector -norm is given by

r

X = max x k ,

(4-12)

the vector's largest component. Subordinate to the vector -norm is the matrix -norm

A = max

i

F aI

GG ij JJ .

Hj K

(4-13)

That is, the matrix -norm is the maximum of the row sums. To see this, let A be an arbitrary

mn matrix and compute

a1k xk

k

a2k x k

r

k

AX =

a mk xk

k

max

i

CH4.DOC

= max

i

aik x k max

k

F axI

GH k ik k JK

(4-14)

F a I max x

GH k ik JK k k

Page 4-5

EE448/528

Version 1.0

= max

i

F a I

GH k ik JK

John Stensby

r

X

i

r

r

AX K X

F a I for which

GH k ik JK

(4-15)

for all X.

Step #2 requires that we find a non-zero X0 for which equality holds in (4-14) and (4-15).

A close examination of these formulas leads to the conclusion that equality prevails if X0 is

defined to have the components

xk

=

=

aik

, a ik 0, 1 k n,

a ik

1,

a ik = 0

(4-16)

(the overbar denotes complex conjugate) where i is the index for the maximum row sum. That is,

in (4-16), use index i for which

aik a jk

k

, i j.

(4-17)

Recall that the vector 2-norm is given by

CH4.DOC

Page 4-6

EE448/528

n

r

X2=

Version 1.0

xi

John Stensby

r r

X, X .

(4-18)

i =1

A 2 = largest eigenvalue of A A .

(4-19)

Due to this connection with eigenvalues, the matrix 2-norm is called the spectral norm.

To see (4-19) for an arbitrary mn matrix A, note that A*A is nn and Hermitian. By

Theorem 4.2.1 (see Appendix 4.1), the eigenvalues of A*A are real-valued. Also, A*A is at least

positive semi-definite since X*(A*A)X = (AX)*(AX) 0 for all X. Hence, the eigenvalues of

A*A are both real-valued and non-negative; denote them as

12 22 32 L 2n 0 .

(4-20)

Note that these eigenvalues are arranged according to size with 12 being the largest. These

eigenvalues are known as the singular values of matrix A.

Corresponding to these eigenvalues are n orthonormal (hence, they are independent)

r r

(A*A)Uk = (k2)Uk , 1 k n.

(4-21)

The n eigenvectors form the columns of a unitary nn matrix U that diagonalizes matrix A*A

under similarity (matrix U*(A*A)U is diagonal with eigenvalues (4-20) on the diagonal).

r r

Since the n eigenvectors U1, U2, ... , Un are independent, they can be used as a basis, and

CH4.DOC

Page 4-7

EE448/528

Version 1.0

John Stensby

n

r

r

X = ck U k ,

(4-22)

k =1

where the ck are X-dependent constants. Multiply X, in the form of (4-22), by A*A by to obtain

n

n

r

r

r

A AX = A A ck U k = c k 2k U k ,

k =1

(4-23)

k =1

which leads to

r 2

r

r r

r

AX = ( AX ) AX = X ( A AX ) =

2

12

F n c Ur I F n c 2 Ur I = n c 2 2

GH k=1 k k JK GH j=1 j j jJK k=1 k k

F n c 2I

GH k=1 k JK

(4-24)

r 2

= 12 X

2

for arbitrary X. Hence, we have completed step#1: we found a constant K = 12 such that

r

r

r

r

AX 2 K X 2 for all X. Step#2 requires us to find at least one vector X0 for which equality

r

r

r

holds; that is, we must find an X0 with the property that AX 0 2 = K X 0 2 . But, it is obvious

that X0 = U1, the unit-length eigenvector associated with eigenvalue 12 , will work. Hence, the

matrix 2-norm is given by A 2 =

The 2-norm is the default in MatLab. Also, it is the default here. From now on, unless

specified otherwise, the 2-norm is assumed: A means A 2 .

Example

A=

LM1 1OP

N2 1Q

CH4.DOC

Page 4-8

EE448/528

Version 1.0

John Stensby

r

r

Find A 2 , and find the unit-length vector X0 that maximizes AX . First, compute the product

2

5 3

.

A A = A TA =

3 2

LM OP

N Q

The eigenvalues of this matrix are 12 = 6.8541 and 22 = .1459; note that ATA is positive definite

symmetric. The 2-norm of A is A 2 = 12 = 2.618 , the square root of the largest eigenvalue of

ATA.

The corresponding eigenvectors are U1 = [-.8507 -.5257] and U2 = [.5257 -.8507]. They

are columns in an orthogonal matrix U = [U1 U2]; note that UTU = I or UT = U-1. Furthermore,

matrix U diagonalizes ATA

L

12

( A A )U = M

MN 0

T

OP L

PQ MN

6.8541

0

0

=

.1459

0

22

r

The unit-length vector that maximizes AX

r

max

AX

r

X =1

OP

Q

r

is X0 = U1, and

= 12 = 6.8541

For mn matrix A we know that

r

r

AX p A p X p

(4-25)

for all X. Now, consider mn matrix A and nq matrix B. The product AB is mq. For q-

dimensional X, we have

CH4.DOC

Page 4-9

EE448/528

Version 1.0

r

r

r

r

ABX p = A ( BX ) p A p BX p A p B p X p .

John Stensby

(4-26)

r

ABX p

Ap Bp

r

Xp

(4-27)

AB p A p B p ,

(4-28)

2-Norm Bound

Let A be an mn matrix with elements aij, 1 i m, 1 j n. Let

max a ij

i, j

denote the magnitude of the largest element in A. Let (i0, j0) be the indices of the largest element

so that

a i j a ij

0 0

(4-29)

for all i, j.

CH4.DOC

Page 4-10

EE448/528

Version 1.0

John Stensby

max a ij A 2

mn max a ij

i, j

(4-30)

i, j

r

AX

2

2

a1k x k

A 2 = max

r

X =1

a2k x k

k

+L+

a mk x k

(4-31)

r

AX .

2

Define X0 as the vector with 1 in the j0 position and zero elsewhere (see (4-29) for definition of i0

and j0). Since

r

r

AX = A 2

AX 0 2 max

r

2

X 2 =1

(4-32)

we have

ai j

0 0

a k j0 2

k

r

max

AX 2 = A 2

r

X =

2

(4-33)

Hence, we have

max a ij A 2

i, j

(4-34)

as claimed. Now, we show the remainder of the 2-norm bound. Observe that

CH4.DOC

Page 4-11

EE448/528

Version 1.0

2

A 2 = max

r

X 2 =1

a1k x k

max

r

X 2 =1

a2k x k

+L+

a1k xk

John Stensby

a mk x k

k

+ a 2 k x k + L + a mk x k

2

(4-35)

a1k 2 + a2k 2 + L + a mk 2

k

mn max a ij .

i, j

i, j

mn max a ij .

i, j

Recall the Triangle inequality for real numbers: = + . A similar result is

valid for the matrix p-norm.

Let A and B be mn matrices. Then

A+Bp A p+ Bp

(4-36)

Let A be an nn non-singular matrix. Can we state conditions on the size(i.e., norm) of

nn matrix E which guarantees that A+E is nonsingular? Before developing these conditions, we

derive a closely related result, the geometric series for matrices. Recall the geometric series

1

= xk , x < 1

1 x k =1

(4-37)

for real variables. We are interested in the matrix version of this result.

CH4.DOC

Page 4-12

EE448/528

Version 1.0

John Stensby

Let F be any nn matrix with F 2 < 1 . Let I denote the nn identity matrix. Then, the

difference I - F is nonsingular and

( I F)1 =

Fk

(4-38)

k=0

with

1

.

( I F)1 2

1 F 2

(4-39)

Proof: First, we show that I - F is nonsingular. Suppose that this is not true; suppose that I - F is

r

r

r

r

singular. Then there exists at least one non-zero X such that (I - F)X = 0 so that X 2 = FX 2 .

r

r

r

r

r

But since FX 2 F 2 X 2 , we must have X 2 = FX 2 F 2 X 2 which requires that F 2 1 ,

a contradiction. Hence, the nn matrix I - F must be nonsingular. To obtain a series for (I - F)-1,

consider the obvious identity

F N Fk I ( I F) = I F

GH k=0 JK

N +1

(4-40)

N

F

I

limit G Fk J ( I F) = I

N H k = 0 K

so that ( I F)

(4-41)

Fk

as claimed. Finally

k=0

CH4.DOC

Page 4-13

EE448/528

( I F)

1

2

Version 1.0

k =0

F2

John Stensby

Fk 2

k =0

(4-42)

k =0

1

1 F 2

as claimed.

In many applications, Theorem 4.3 is used in the form

( I F )

Fk k

k =0

for < 1 / F 2 ,

(4-43)

where is considered to be a "small" parameter. Next, we generalize Theorem 4.3 and obtain a

result that will be used in our study of how small errors in A and b influence the solution of the

Theorem 4-4

Assume that A is nn and nonsingular. Let E be an arbitrary nn matrix. If

= A 1E 2 < 1,

(4-44)

2

( A + E )1 A 1 2

E 2 A 1

2

1

(4-45)

CH4.DOC

Page 4-14

EE448/528

Version 1.0

John Stensby

(4-46)

( I F)1 2

1 F 2

1

.

1

(4-47)

From (4-46) we have (A + E)-1 = (I - F)-1A-1; with the aid of (4-47), this can be used to write

( A + E)

1

2

A 1

2 .

(4-48)

(4-49)

by A + E to see this matrix identity. Finally, take the norm of (4-49) to obtain

( A + E )1 A 1

= A 1E( A + E )1

(4-50)

A

1

2

E 2 ( A + E)

1

2

( A + E)

1

2

A 1

1

2

E2

A 1

2

2

E2

(4-51)

as claimed.

CH4.DOC

Page 4-15

EE448/528

Version 1.0

John Stensby

We will use Theorem 4.4 when studying the sensitivity of the linear equation AX = b.

That is, we want to relate changes in solution X to small changes (errors) in both A and b.

CH4.DOC

Page 4-16

- Unstable&VeryUnstableUploaded bydomeleu
- SOLUTION OF EQUATION AX + XB = C BY INVERSION OF AN M ×M OR N × N MATRIXUploaded byPeet89
- Linear Algebra MATLABUploaded bySharmistha Mukherjee
- Matrix ReviseUploaded byFatmah El Wardagy
- Saad Y., Iterative Methods for Sparse Linear SystemsUploaded byTopografia Mecanica
- Operator equationUploaded byFotis Kasolis
- A. Huseynov, G.S. GuseinovUploaded bylaputamadre1
- MatricesUploaded byvignanaraj
- A Rank-1 Update AlgorithFastSolvKernel Foleyamp; SammonDiscriminant Vect-DkzUploaded by50_BMG
- Zakpoane PaperUploaded byZuhri Muin
- Matlab IntroUploaded byadnan
- Study Notes on Matrix Algebra PrepLadderUploaded bymanikantamnk11
- 1-9Uploaded byDita Pramesti
- Matrix NormUploaded byDivya Tejaswini
- HW12Uploaded byCody Sage
- Math for Machine LearningUploaded byjoekowalsky
- Non SingularUploaded byamera_na
- Twodof Base Disp Modal RevAUploaded byTamara Knox
- Math2081 2010C Midterm SolUploaded byTriet Nguyen
- FDLFUploaded byvaithy1990
- Standard Akademik f5Uploaded byRisma Robin
- FACE AND PALMPRINT RECOGNITION SYSTEMUploaded byPrabhakarSingh
- mth401Uploaded byhumza8081
- BeelenV88aUploaded byGufran Ansari
- Tam ProbabilityLinearAlgebraReviewUploaded bykyawmoesoe
- Tamas Szidarovszky, Attila G. Csaszar and Gabor Czako- On the efficiency of treating singularities in triatomic variational vibrational computations. The vibrational states of H3^+ up to dissociationUploaded byTreaxmeA
- NFEM.Ch05.pdfUploaded byAssefa Nigussie
- [Helmke_U.,_Moore_J.]_Optimization_and_dynamical_s(BookSee.org).pdfUploaded byatif313
- Linear AlgebraUploaded bySivanathan Anbu
- Computer ScienceUploaded byAnurag Garg

- LU DecompositionUploaded byShyam Prakash
- ASIO4ALL v2 Instruction ManualUploaded byAndrew Dprssn GavriLuv
- Correction TD3 4Uploaded byCédric Mzt
- Ecole de Musique Mr MuscatUploaded byCédric Mzt
- La Kabylie et les Coutumes kabyles 1/3, par Hanoteau et Letourneux, 1893Uploaded byTamkaṛḍit - la Bibliothèque amazighe (berbère) internationale
- gestionUploaded byCédric Mzt
- AnalyseUploaded byCédric Mzt
- PermutationsUploaded byCédric Mzt
- colles 14-09-10 (1)Uploaded byCédric Mzt
- Cours Proba-stats Partie 1Uploaded byCédric Mzt
- Fiche ViergeUploaded byCédric Mzt
- Fiche ViergeUploaded byCédric Mzt
- 02 Series Numeriques DemonstrationsUploaded byanas_fr01
- complément éléments propres endomorphismeUploaded byCédric Mzt
- Test 12Uploaded byCédric Mzt

- ￼From Traditional Set Theory – that of Cantor, Hilbert, Gödel, Cohen – to Its Necessary Quantum ExtensionUploaded byegbelaga
- Approximata value_RIP_V_Day_1.pdfUploaded bymathigangadhar
- Design, Implementation, And Simulation of A Fuzzy controllerUploaded byAbcvdgtyio Sos Rsvg
- Estimating Entropy of Data Streams Using Compressed CountingUploaded byterminatory808
- CUDNN Library (1)Uploaded byAriel Lenis
- Error Detection Recovery Compiler Design PDFUploaded byPaul
- LAB (Flowmeter Demonstration)Uploaded byNurul Ain
- When Should You BypassUploaded byAnonymous 1A7olH
- The Dyadic Adjustment ScaleUploaded bymars1001
- BarfieldUploaded byFritzie Mae Inoc Faller
- (Grundlehren der mathematischen Wissenschaften 258) Joel Smoller (auth.)-Shock Waves and Reaction—Diffusion Equations-Springer US (1983).pdfUploaded bylig
- 3 SpinsUploaded byHaley Wang
- RPP_05_BDA37101 TERKINI 2014Uploaded byAina Syazwana Zulkefli
- Algorithms Notes for ProfessionalsUploaded byconrado
- coUploaded bytbijle
- The Cambridge Handbook of Artificial IntelligenceUploaded bykim
- C03-The Cellular ConceptUploaded byArjun Mukhopadhyay
- Nuclear Physics - HandoutsUploaded byGerry Lou
- STID3024 Week6,7 InteractionDiagramUploaded byPei Ting
- Fourier IntegralUploaded byAbhiram Giri
- amyloseUploaded byjohneiver
- 3rd grade numbel jumble 2016-2017 - student listUploaded byapi-324884203
- LAB CADENCEUploaded bydavilloo
- STS AssignmentUploaded byAdri versouisse
- Dynamic Analysis Using FEAUploaded byNallathambi
- Research Paper on Consumer IdentityUploaded byMirza Junaid
- Int Physics Olympiad Questions 1_2_3Uploaded byrock
- Textbook of Linear Algebra by Jim HefferonUploaded byCSF511112
- Documents.mx Computer Program for Determination of Geometrical Properties of Thin Walled 577b0326dd543Uploaded byMarcelo Gimenez Magalhães
- Ch 14 Work StudyUploaded byErlinda Empleo