Semidenite matrices & Convex functions

Rudi Pendavingh
Eindhoven Technical University

Optimization in Rn , lecture 7

Rudi Pendavingh (TUE)

Semidenite matrices & Convex functions

ORN7

1 / 17

Semidenite matrices
A symmetric matrix A is positive semidenite (PSD) if and only if x t Ax 0 for all x Rn .

Theorem
Let A be a symmetric matrix. The following are equivalent:
1 2 3

A is PSD, i.e. x t Ax 0 for all x Rn . all eigenvalues of A are nonnegative. A = Z t Z for some real matrix Z .

Corollary
Let p R[X1 , . . . , Xn ] be a homogeneous quadratic polynomial. Then p (x1 , . . . , xn ) 0 for all x1 , . . . , xn R if and only if
2 2 p = s1 + + sn

for some si R[X1 , . . . , Xn ]

Rudi Pendavingh (TUE) Semidenite matrices & Convex functions ORN7 2 / 17

Recognizing PSD matrices

Denition
symmetric matrix operations on a matrix are:
1 2

multiplying both the i -th row and i -th column by = 0 swapping the i -th and j -th column; and swapping the i -th and j -th row adding i -th column to j -th column and adding i -th row to j -th row

A = B : B is obtained from A by zero or more symmetric matrix operations.

Lemma
A = B if and only if B = Y t AY for some invertible Y .

Lemma
Let A, B be symmetric. If A = B , then A is PSD B is PSD.
Rudi Pendavingh (TUE) Semidenite matrices & Convex functions ORN7 3 / 17

Recognizing PSD matrices

To determine whether A is PSD, compute a diagonal matrix D = A.

Example
A= 1 2 1 0 = D. = 0 1 2 3

D is not PSD, hence A is not PSD.

Example
1 0 0 1 2 1 1 0 0 A= 2 5 1 = 0 1 3 = 0 1 0 = D . 1 1 12 0 3 11 0 0 2 D is PSD, hence A is PSD.

Rudi Pendavingh (TUE)

Semidenite matrices & Convex functions

ORN7

4 / 17

Positive denite matrices

A matrix A is positive denite (PD) if x t Ax > 0 for all nonzero x Rn .

Lemma
A is PD A is PSD and det(A) = 0.

Theorem
Let A be a symmetric matrix. The following are equivalent:
1 2 3

A is PD, i.e. x t Ax > 0 for all nonzero x Rn . all eigenvalues of A are positive. A = Z t Z for some real matrix Z so that det(Z ) = 0.

Lemma
Let A, B be symmetric. If A = B , then A is PD B is PD.

Rudi Pendavingh (TUE)

Semidenite matrices & Convex functions

ORN7

5 / 17

Further characterizations

Let A be a square matric and let I {1, . . . , n}. Then AI denotes the restriction of A to the rows and columns indexed by I .

Lemma
Let A be an n n matrix. Then A is PSD det(AI ) 0 for all I {1, . . . , n}.

Lemma
Let A be an n n matrix. Then A is PD det(AI ) > 0 for I = {1}, . . . , {1, . . . , n}.

Rudi Pendavingh (TUE)

Semidenite matrices & Convex functions

ORN7

6 / 17

Ger sgorins Theorem

Given a complex n n matrix A, we put p :=
j =p

|apj | for p = 1, . . . , n.

Theorem (Ger sgorin, 1931)

Let A be a complex n n matrix and let be an eigenvalue of A. Then | app | p for some p {1, . . . , n}.

Proof.
Let x Cn be a nonzero vector such that x = Ax . Let p be such that |xp | = maxi |xi |. We have xp =
n j =1 apj xj ,

hence ( app )xp =

j =p apj xj |

j =p apj xj .

Taking norms, | app ||xp | = | Dividing by |xp |, the Theorem follows.

j =p

|apj ||xj | p |xp |.

For each p , we have the Ger sgorin disk { C | | app | p }. The Theorem states that each eigenvalue is in one of the Ger sgorin disks.
Rudi Pendavingh (TUE) Semidenite matrices & Convex functions ORN7 7 / 17

Convex and concave functions

Denition
A function f : Rn R is convex if Dom(f ) is convex, and f (x ) + (1 )f (y ) f (x + (1 )y ), for all x , y Dom(f ) and for all [0, 1].

Denition
f is concave if f is convex.

Lemma
f is convex and concave if and only if f is ane.

Rudi Pendavingh (TUE)

Semidenite matrices & Convex functions

ORN7

8 / 17

Norms
Denition
A function f : Rn R is a norm if f (x ) 0 for all x Rn f (x ) = 0 if and only if x = 0 f (x ) = ||f (x ) for all x Rn , R f (x + y ) f (x ) + f (y ) for all x , y Rn

Theorem
Let f : Rn R be a norm. Then f is convex.

Lemma
Let f : Rn R be a convex function. Then {x Rn | f (x ) } is a convex set for any R. So the norm ball {x Rn | x 1} is convex for any norm . .
Rudi Pendavingh (TUE) Semidenite matrices & Convex functions ORN7 9 / 17

Denition
The epigraph of a function f : Rn R is epi(f ) := {(x , t ) Rn+1 | x Dom(f ), t f (x )}.

Lemma
f is a convex function if and only if epi(f ) is a convex set.

Denition
A subgradient of f at x is a row vector w such that f (y ) f (x ) w (y x ) for all y .

Theorem
f is convex Dom(f ) is convex, and there exists a subgradient of f at each x in the interior of Dom(f ).
Rudi Pendavingh (TUE) Semidenite matrices & Convex functions ORN7 10 / 17

Operations that preserve convexity

Lemma
If f : Rn R is convex, then the following functions are convex as well: x f (x ), for any > 0 x f (x + t ), for any xed t Rn x f (Ax ), for any n m matrix A

Lemma
If f1 , . . . , fm : Rn R are convex, then the following functions are convex as well: x max{f1 (x ), . . . , fm (x )} x f1 ( x ) + + fm ( x )

Rudi Pendavingh (TUE)

Semidenite matrices & Convex functions

ORN7

11 / 17

Dierentiable convex functions

Denition
f f The gradient of f : Rn R at x is f (x ) := ( x1 (x ), . . . , xn (x )).

Lemma
If w is a subgradient of f at x , and f is dierentiable at x , then w = f (x ).

Theorem (First-order condition for convexity)

A dierentiable function f is convex if and only if f (y ) f (x ) + f (x )(y x ) for all x , y Dom(f ). Note: x is a minimizer of f if and only if 0 is a subgradient of f at x .
Rudi Pendavingh (TUE) Semidenite matrices & Convex functions ORN7 12 / 17

Twice dierentiable convex functions

Denition
The Hessian of a function f : Rn R at x is the matrix 2f 2f (x ) x 2 (x ) x1 1 xn . . . . . 2 f (x ) := . . 2 2 f f ( x ) ( x ) xn x1 x 2
n

Theorem (Second-order condition for convexity)

Let f be a twice dierentiable function. Then f is convex Dom(f ) is convex and 2 f (x ) is positive semidenite for all x in the interior of Dom(f ).

Rudi Pendavingh (TUE)

Semidenite matrices & Convex functions

ORN7

13 / 17

Quadratic functions

Example
Let f : x f (x )
x t Qx 2 + px , = xtQ + p

where Q is a symmetric matrix, p a row vector. for all x

2 f (x )

= Q for all x

f is convex if and only if Q is PSD if f is convex, then f (y ) = min{f (x ) | x Rn } if and only if 0 = f (y ) = y t Q + p Note: f is convex if and only if the second-order approximation of f is convex everywhere.

Rudi Pendavingh (TUE)

Semidenite matrices & Convex functions

ORN7

14 / 17

Some convex functions

Example (Functions of one variable)

x e ax is convex on R, for any a R x x a is
convex on {x R | x > 0} when a > 1 or a < 0 concave on {x R | x > 0} when 0 a 1

x log(x ) is concave on {x R | x > 0} x x log(x ) is convex on {x R | x > 0}

Rudi Pendavingh (TUE)

Semidenite matrices & Convex functions

ORN7

15 / 17

More convex functions

Example (Functions of several variables)

(x , y ) x 2 /y is convex on {(x , y ) R2 | y > 0} (x1 , . . . , xn ) max{x1 , . . . , xn } is convex on Rn (x1 , . . . , xn ) log(e x1 + + e xn ) is convex on Rn (x1 , . . . , xn ) (
1 n n i =1 xi )

is concave on {x Rn | x > 0}

X log(det(X )) is concave on {X | X is PSD, det(X ) = 0}

Rudi Pendavingh (TUE)

Semidenite matrices & Convex functions

ORN7

16 / 17

The function f : X log(det(X )) is convex

Proof.
It suces to show that for any PD X and symmetric Y , the function g : t log(det(X + tY )) is convex. As X is PD, there exists a PD matrix Z so that Z 2 = X . Hence g (t ) = log(det(ZZ )) log(det(Z 1 (X + tY )Z 1 )) = = log(det(X )) log(det(I + tY )) where Y = Z 1 YZ 1 . Let 1 , . . . , n be the eigenvalues of Y . Then det(I + tY ) = i (1 + t i ), hence g (t ) = log(det(X ))
i

log(1 + t i ).

This is a sum of a constant and convex funtions t log(1 + t ).

Rudi Pendavingh (TUE) Semidenite matrices & Convex functions ORN7 17 / 17