Approximating the Matrix Exponential: In partial completion of the course Numerical Linear Algebra M383E

Joshua Lee Fisher and Nicholas Christian December 12, 2005

Abstract The goal of this project is to compare the pros and cons of certain series methods and ordinary dierential equation (ODE) methods for approximating the matrix exponential. As well as discuss which methods are better for symmetric and positive denite matrices.

Introduction

Many models of physical application involve the calculation of the matrix exponential. We dene it to be the convergence of the power series

e =
i=0

Ai i!

(1)

where A is an m m real matrix . Several tools are available to compute eA such as truncated series, ordinary dierential equations, polynomials, and matrix decompositions. However, in this project we are going to focus on the advantages and disadvantages of some truncated series methods and ordinary dierential equations (ODE) methods. In 2 we take a look at the Taylor Series, Pad e Approximation, and Scaling and Squaring techniques. Then in 3 we look at solutions to the 1

linear constant coecient ordinary dierential equation x (t) = Ax(t). The At solution to this evolution equation is x = e . Thus when t = 1 we have the desired approximation eA . Next in 4 we look at the better methods for calculating the matrix exponential when dealing with a symmetric matrix and a positive denite matrix. In this paper we will be using || || to mean the 2-norm,
n

||x|| =
i=1

|x i |

1 2

For a matrix A we use the induced 2-norm.

Series Methods

In this section we will describe the various series methods to computing the matrix exponential which include; Taylor series, Pad e approximations and scaling and squaring.

2.1

Taylor Series

From (1) a truncated Taylor series is an obvious choice for an algorithm (Algorithm 1). However there are several short comings with this procedure, ie. roundo error and computation time. For those concerned with eciency, where to truncate the series is a problem. For example, let 2 3 4 A = 1 5 2 . 7 0 1 Then it takes 41 terms to calculate eA using Algorithm 1. 773.0437 884.8837 739.2448 eA = 321.8790 424.2469 325.5624 . 831.0482 925.2604 786.5025 A similar problem is that as ||A|| increases, the Taylor series method looses accuracy while taking longer to calculate. This is evident from the 2

graphs in 4. Interestingly, the problem with the Taylor series accuracy is not the truncation of the series but the roundo error from calculating the terms. In the appendix of [7], the author presents an iterative algorithm for calculating eA using a Taylor series as well as a method for determining where to truncate the series to get so many signicant digits. In order to compute the Taylor expansion our code has been written in the Matlab1 environment, therefore the algorithms we will discuss will be geared toward the terminology used in Matlab . Algorithm 1 describes the Taylor series expansion of eA . The input is the m m matrix A. If A is the only input then the algorithm will expand until it reaches a specied level of convergence. One can also input the number of terms they want to expand to, i.

2.2

Pad e Approximation

Using a Pad e approximation is very similar to using a Taylor series except for one major dierence. The purpose of a Taylor series is to expand a function as a power series, whereas a Pad e approximation expands a function as the ratio of two power series. For an overview of Pad e approximates consult [6]. Calculating the matrix exponential using Pad e Approximation is dened by, Rpq (A) = [Dpq (A)]1 Npq (A), (2) where
p

Npq (A) =
j =0 q

(p + q j )!p! Aj (p + q )!j !(q j )! (p + q j )!q ! (A)j . (p + q )!j !(q j )!

Dpq (A) =
j =0

Notice the similarity with the Taylor series, if q = 0 then (2) would be a Taylor series. Algorithm 2 is the Pad e approximation of eA . The inputs are A, p and q . This method works best when p = q .
Matlab is a registered trademark of The MathWorks, Inc., 3 Apple Hill Drive, Natick, MA 01760-2098, USA, tel 508-647-7000, fax 508-647-7001, info@mathworks.com, http:www.mathworks.com
1

Algorithm 1 (A, i), Taylor Series Expansion of eA , A Cmm expA1 = eye(m) %% Initializes an m m identity matrix. expA2 = zeros(m) %% Initializes an m m matrix of zeros. %% expA2 is used to check convergence within specied tolerance. tol = 1015 if nargin == 1 then %% when the input to the algorithm is only A k=1 while ||expA2 expA1 ||2 > tol do k expA1 = expA1 + A k! k =k+1 k expA2 = expA1 + A k! end while expA1 = expA2 else %% when input includes the number of terms in expansion for k = 1 to i do k expA1 = expA1 + A k! end for end if return expA1 return k The Pad e algorithm has some advantages over the Taylor series. In particular, Pad e can obtain the same accuracy as Taylor in signicantly less time, see 4. A draw back of Pad e is that the algorithm performs poorly as ||A|| increases. From Higham [8] it was seen that the Pad e approximation are more ecient when using p = q , ie. the diagonal entries of the Pad e table in [6]. Like the Taylor series there is the question of where to terminate the series, what are the appropriate values of p. Most articles say that 8 or 6 terms will give the best approximation, however [8] suggests that 13 terms gives the best approximation. For further error analysis see Van Loan [2].

Algorithm 2 (A, p, q ), Pad e Approximation of eA , A Cmm N =zeros(m) D =zeros(m) temp = 0 for j = 0 to p do p+q j )!p! N = N + (p(+ Aj q )!j !(pj )! end for temp = 0 for j = 1 to q + 1 do (p+q j )!)q ! D = D + (p (A)j +q )!j !(q j )! end for R = D1 N return R

2.3

Scaling and Squaring

The main problem with the Taylor Series and Pad e approximation is that the accuracy decreases and eciency increases as ||A|| increases. To over come this problem we use the following identity, eA = (eA/n )n (3)

Typically, n is chosen to be the smallest power of two such that ||A|| < 1. Algorithm 3 shows the implementation of 3. Algorithm 3 is the scaling and squaring method. The inputs are @f name which is the method we want to evaluate eA with, along with A and j where n = 2j . Algorithm 3 (fname,A,j ,varargin), Scaling and Squaring a Matrix, e( n )n , A Cmm %% fname is the function for computing eA , enter as @fname. %% varargin are the other input arguments for fname.m n = 2j A= A %% Scales A by 1/n n S =feval(fname,A,varargin{:}) A S = S n %% Squares e n by n e algorithm, To approximate (eA/2 )2 using p terms in the diagonal Pad 5
j j A

)m3 ops. Similarly to approximate (eA/2 )2 using k terms it takes (p + j + 1 3 in the Taylor series algorithm, it takes (k + j + 1 )m3 ops. 3 In [4], Ward provides relative error bounds for a diagonal Pad e approximation algorithm that uses scaling and squaring. Ward also provides a statistical roundo error analysis of the same algorithm in [5].

Ordinary Dierential Equation Methods

The ordinary dierential equation method is motivated by the solution to the equation x (t) = Ax(t), 0 t 1, (4) where A is a constant coecient matrix and x is a vector function with respect to t. Then the solution to (4) is x = eAt . Therefore at t = 1 we have the solution x = eA . We will look at three methods for solving (4) then compare the solution for symmetric and positive denite matrices in 4.1 and 4.2 respectively. The three methods we will examen are: General Purpose ODE Solver, Sti solver ODE Methods, and Multistep ODE Solver.

3.1

General Purpose ODE and Sti Solvers

The general purpose ODE solver uses the Runge-Kutta-Fehlberg Method to solve (4) (Algorithm 4)2 . This gives us a fth order solution with fourth order accuracy. For a review of the Runge-Kutta methods see [3]. Algorithm 4 is the general ODE solver Method. Consider the ordinary dierential equation x = Ax whos solution is x = etA . Then when t = 1 we have the solution to eA . Therefore in Algorithm 4 we use the Runge-Kutta = AX . Algorithm method to solve the system of dierential equations X 5 solves this system using the Adams-Bashforth-Moulton predictor-corrector method. The predictor-corrector method can be seen in Algorithm 6. We have compared the Runge-Kutta-Fehlberg method with a sti solver using the Matlab function ode23s3 . This is a lower order method with a third order solution and second order accuracy. However it can be seen in 4 that the sti solver had the worst eciency taking over ve minutes to solve a 100 100 matrix with no real gain in accuracy.
In this algorithm we use the Matlab function ode45 to solve the ODE. For this method we have used the same algorithm, Algorithm 4, only changing the solver to ode23s.
3 2

Algorithm 4 General ODE Solver Method = AX where X = etA , A Cmm %% Solves X tspan = [0, 1] I =eye(m) options = [ ] for k = 1 to m do x0 = I:,k %% the k th column of the identity [t, x] =ode45(@AX, tspan, x0 , options, A) = AX %% @AX is the function X [row, col] =size(x) X:,k = xT %% solution at t = 1 row,: end for return X

3.2

Multistep ODE Solver (Adams-Bashforth-Moulton)

Here we use the so called predictor-corrector method to solve (4). This method uses an explicit formula to obtain an initial estimate then an implicit formula to obtain more accurate results. Algorithm 6 describes the AdamsBashforth-Moulton ODE solver. It uses an open Adams-Bashforth scheme for the predictor step and a closed Adams-Moulton scheme for the corrector step. Algorithm 6 uses a xed time step of 0.04. This accounts for its poor performance compared to the adjustable time step incorporated in the above methods (which can be seen in 4). In order to solve the matrix equation (4) we use Algorithm 5.

Results For Two Types of Matrices

Here we show the results for two types of matrices; symmetric and positive denite. The computations were run using Matlab version 6.5 on a Dell Inspiron 600m laptop with a Intel Pentium M processor 1.80GHz. We have performed several numerical experiments for varying matrices, A Rmm . In several of the graphs comparing accuracy and eciency we have plotted the methods separately due to varying scales.

Algorithm 5 (A), (Adams-Bashforth-Moulton) Exponential Solver = AX where X = etA , A Cmm %% Solves X tspan = [0, 1] I =eye(m) options = [ ] for k = 1 to m do x0 = I:,k %% the k th column of the identity [t, x] =odepc(@AX, tspan, x0 , options, A) %% odepc is the function in Algorithm 6 = AX %% @AX is the function X [row, col] =size(x) X:,k = xT %% solution at t = 1 row,: end for return X

4.1

Symmetric Matrices

An m m matrix A is symmetric if aij = aji for all 1 i m, 1 j m. Since A is a real symmetric matrix we are guaranteed real eigenvalues. Therefore in order to compute eA when A is symmetric we use the eigenvalue decomposition of A as a reference to examine how well the various methods work. The eigenvalue decomposition of A is, A = V V 1 . Then by using (5) we can now calculate eA ,

(5)

=
k=0

1 (V V 1 )k k! 1 V k V 1 k!

= V e V 1 . For = diag {1 , , m }, e = diag {e1 , , em }. That is eA = V diag {e1 , , em }V 1 . (6)

k=0

Therefore to compute eA you only need to calculate the exponential of the entries on the diagonal of . 8

Accuracy, A Cm m, m=25 0.14 0.12 0.1 Relative Error 0.08 0.06 0.04 0.02 0 0 2 4 6 Increasing ||A|| 8 10
0 0

Accuracy of Pade, A Cm m, m=25 1500

Taylor RungeKutta45 RungeKutta23 stiff PredictorCorrector

Relative Error

1000

500

4 6 Increasing ||A||

10

(a)

(b)

Figure 1: This gure shows the accuracy of varying 25 25 symmetric matrices, A. We see that all the methods do well with the exception of Pad e, which failed for ||A|| 7 ( in graph (b)) and predictor-corrector (in graph (a)). Figure 1 we show the accuracy of varying 25 25 symmetric matrices, A. We see that all the methods do well with the exception of Pad e, which failed for ||A|| 7 (in graph (b)) and predictor-corrector (in graph (a)). We see the eciency for Figure 1 plotted in Figure 2. This gure shows the eciency of these methods. On the x-axis we have the increasing ||A|| and on the y-axis we have the time in seconds. We see the most ecient to be the Pad e method (red line) and the least ecient to be the sti solver (black line), which takes up to 5 seconds of computation time. This illustrates the general trend of how fast each method is computed. In Figure 3 we show the accuracy for varying 50 50 symmetric matrices. For these matrices we see similar results as in Figure 1 with the exception that Pad e starts to fail for ||A|| 5. Again we see the same trend in eciency for Figure 3. The slowest method again appears to be the sti solver which can be seen in Figure 4 which took up to 30 seconds in the case where A Rnn , n = 50. Finally we look at 100 100 symmetric matrices. In Figure 5 we see 9

Efficency, A C 0.35 0.3 Time Efficency (sec) 0.25 0.2 0.15 0.1 0.05 0 0 2 Pade Approximation Taylor Series RungeKutta45 PredictorCorrector

m m

Efficency, RungeKutta23 stiff, A Cm m, m=25 5

, m=25

4 Time Efficency (sec)

4 6 Increasing ||A||

10

0 0

4 6 Increasing ||A||

10

(a)

(b)

Figure 2: This gure shows the eciency of these methods. On the x-axis we have the increasing ||A|| and on the y-axis we have the time in seconds. We see the most ecient to be the Pad e method (red line) and the least ecient to be the sti solver (black line), which takes up to 5 seconds of computation time. that for large matrices predictor-corrector fails quickly followed by Pad e and Taylor. Figure 6 shows the sti solver took the longest time coming in at approximately six minutes. This was followed by Taylor at four seconds then Runge-Kutta, predictor-corrector and the fastest being Pad e. Here we have shown that in general the Runge-Kutta and Taylor methods consistently show the best accuracy where as the Pad e approximation performs fastest.

4.2

Positive Denite Matrices

In this section we will investigate the results for the various matrix exponential methods for positive denite matrices. By denition A is Hermitian positive denite if all eigenvalues are positive.

10

Accuracy, A Cm m, m=50 0.2 Taylor RungeKutta45 RungeKutta23 stiff PredictorCorrector

Accuracy of Pade, A Cm m, m=50 2.5

2 Relative Error

0.15 Relative Error

1.5

0.1

0.05

0.5

0 0

6 8 Increasing ||A||

10

12

0 0

6 8 Increasing ||A||

10

12

(a)

(b)

Figure 3: This gure shows the accuracy of varying 50 50 symmetric matrices, A. We see that all the methods do well with the exception of Pad e, which starts to fail for ||A|| 5 (in graph (b)) and predictor-corrector (in graph (a)). In order to have a comparison for the various methods we have constructed a positive denite matrix with a diagonal decomposition in particular, A = P DP 1 , (7) where D is a diagonal matrix. Now we use (7) to calculate eA ,

eA =
k=0

1 (P DP 1 )k k! 1 P Dk P 1 k!

=
k=0

= P e D P 1 .
d For D = diag {d1 , , dn }, eD = diag {ed 1 , , en }. That is

eA = P diag {ed1 , , edn }P 1 . 11

(8)

Efficency, A Cm m, m=50 1 Pade Approximation Taylor Series RungeKutta45 PredictorCorrector

Efficency, RungeKutta23 stiff, A Cm m, m=50 30 25 Time Efficency (sec) 20 15 10 5 0 0

0.8 Time Efficency (sec)

0.6

0.4

0.2

0 0

6 8 Increasing ||A||

10

12

6 8 Increasing ||A||

10

12

(a)

(b)

Figure 4: Eciency of 50 50 symmetric matrices, A. In gure (a) the slowest method is Runge-Kutta (green line). The next slowest is predictorcorrector (cyan line) then Taylor series (blue line) and the fastest being Pad e approximation (red line). In (b) we see the worst method according to efciency to be the sti solver which took up to 30 seconds in the 50 50 case. Therefore to compute eA you only need to calculate the exponential of the entries on the diagonal of D. We rst looked at a positive denite 25 25 matrix, Figure 7 and 8. Here we see that predictor-corrector is the least accurate method while the Taylor series and Pad e approximation provide the best accuracy. We also see that the sti solver is the slowest method, while Pad e is the quickest. Now consider a 50 50 positive denite matrix, Figure 9 and 10. This case is similar to the last one. The predictor-corrector method continues to be the least accurate method and Taylor series is the most accurate. Likewise, the sti solve was the least ecient and the Pad e approximation was the fastest. Next, consider a positive denite 100 100 matrix, Figure 11 and 12. As before predictor-corrector is the least accurate and the Taylor series is the most accurate. Moreover the sti solve is the slowest while Pad e is the quickest. 12

Accuracy, A Cm m, m=100 0.4 0.35 0.3 Relative Error 0.25 0.2 0.15 0.1 0.05 0 0 5 10 Increasing ||A|| 15 20 Taylor RungeKutta45 RungeKutta23 stiff PredictorCorrector

Accuracy of Pade, A Cm m, m=100 3 2.5 Relative Error 2 1.5 1 0.5 0 0

10 Increasing ||A||

15

20

(a)

(b)

Figure 5: This gure shows the accuracy of varying 100 100 symmetric matrices, A. We see that all the methods start to blow up when norm of A increases. The worst being Pad e, which starts to fail for ||A|| 5.

Conclusions

It is clear from all of the previous experiments both symmetric matrices and positive denite matrices that the Pad e approximation method is the most ecient and is very accurate when the ||A|| is less than one. It can be expected that all methods will be more accurate when using the scaling and squaring technique. Therefore, from these experiments we conclude that the best method for computing eA would be a diagonal Pad e approximation method that utilizes scaling and squaring.

References
[1] Cleve Moler and Charles Van Loan, Nineteen Dubious Ways to Compute the Exponential of a Matrix, Twenty-Five Years Later, SIAM Review Vol.45 No.1, pp.3-49, 2003.

13

Efficency, A Cm m, m=100 4 3.5 Time Efficency (sec) 3 2.5 2 1.5 1 0.5 0 0 5 10 Increasing ||A|| 15 20 Pade Approximation Taylor Series RungeKutta45 PredictorCorrector

Efficency, RungeKutta23 stiff, A Cm m, m=100 350 300 Time Efficency (sec) 250 200 150 100 50 0 0 5 10 Increasing ||A|| 15 20

(a)

(b)

Figure 6: Eciency of 100 100 symmetric matrices, A. In gure (a) the slowest method is Taylor series (blue line). The next slowest is Runge-Kutta (green line) then predictor-corrector (cyan line) and the fastest being Pad e approximation (red line). In (b) we see the worst method according to eciency to be the sti solver which approaches six minutes in the 100 100 case. e [2] Charles Van Loan, On the Limitation and Application on Pad Approximation to the Matrix Exponential Pad e and Rational Approximation Theory and Application, Academic press, pp.439-448, 1977. [3] J.H.Mathews and K.D. Fink, Numerical Methods using Matlab, Third Edition, Prentice-Hall, 1999. [4] R.C.Ward, Numerical Computation of the Matrix Exponential with Accuracy Estimate, SIAM Journal of Numerical Analysis, Vol.14 No.4 pp.600-610, 1977. e Algorithm for [5] R.C.Ward, Statistical Roundo Error Analysis of Pad Computing the Matrix Exponential Pad e and Rational Approximation Theory and Application, Academic press, pp. 449-460, 1977.

14

x 10

Accuracy, A Cm m, m=25 Pade RungeKutta23 stiff PredictorCorrector

Relative Error 2.5

x 10

Accuracy, A Cm m, m=25 Taylor RungeKutta45

4 Relative Error

1.5

0.5

0 0

3 4 Increasing ||A||

0 0

3 4 Increasing ||A||

(a)

(b)

Figure 7: For positive denite 25 25 matrices we see that the accuracy is worst for predictor-corrector method (cyan line is (a)) and best for Taylor (blue line in (b)) and Pad e approximations (red line in (a)). [6] G. A. Baker, Jr., Essentials of Pad e Approximants, Academic Press, 1975. [7] M.L. Liou, A Novel Method of Evaluating Transient Response, Proceedings of the IEEE, Vol. 54, No. 1, pp.20-23, 1966. [8] Higham, Nicholas J., The Scaling and Squaring Method for the Matrix Exponential Revisited, SIAM J. Matrix Anal. Appl. Vol.26 No.4, pp.11791193, 2005.

15

Efficency, A Cm m, m=25 0.14 0.12 Time Efficency (sec) 0.1 0.08 0.06 0.04 0.02 0 0 1 2 3 4 Increasing ||A|| 5 6 Pade Approximation Taylor Series RungeKutta45 PredictorCorrector

Efficency, RungeKutta23 stiff, A Cm m, m=25 0.5

0.4 Time Efficency (sec)

0.3

0.2

0.1

0 0

3 4 Increasing ||A||

(a)

(b)

Figure 8: Eciency of 25 25 positive denite matrices, A. In gure (a) the slowest method is Runge-Kutta (green line). The next slowest is predictorcorrector (cyan line) then Taylor series (blue line) and the fastest being Pad e approximation (red line). In (b) we see the worst method according to eciency to be the sti solver which took 0.5 seconds in the 25 25 case.

16

Algorithm 6 (A, n, h), Multi-step (Adams-Bashforth-Moulton) Solver %% Solves x = ax where x = eat , a Cmm , t [0, 1] Rd %% n is the order of the method %% h is the step size %% Initialize the coecients of the nth order method [A, B, M ]=abmcoef(n) %% coecients declared in abmcoef x=[] x = x0 %% Initial guess of rst n terms using Euler Method. for j = 1 to n do x:,j +1 = x:,j + h feval(fname,tj ,x:,j ,varargin{:}) end for f =zeros(m, n) for i = 1 to n do f:,i =feval(fname,ti ,x:,i ,varargin{:}) end for for k = n to d 1 do %% Predictor step h p = x:,k + A (Bf ) n f new = [ ] for i = 2 to n do f new = [f new, f:,i ] end for f = [f new, feval(fname,tk+1 ,p,varargin{:})] %% Corrector step h x:,k+1 = x:,k + A (M f ) n f:,n =feval(f name, tk+1 , x:,k+1 , varargin{:}) end for

17

3.5 3 2.5 Relative Error 2 1.5 1 0.5

x 10

Accuracy, A Cm m, m=50 Pade RungeKutta23 stiff PredictorCorrector Relative Error 1.2 1 0.8 0.6 0.4 0.2

x 10

Accuracy, A Cm m, m=50 Taylor RungeKutta45

0 0

10 15 Increasing ||A||

20

25

0 0

10 15 Increasing ||A||

20

25

(a)

(b)

Figure 9: Here we see similar results to gure 7. For positive denite 50 50 matrices we see that the accuracy is worst for predictor-corrector method (cyan line is (a)) and best for Taylor (blue line in (b)) and Pad e approximations (red line in (a)).

18

Efficency, A Cm m, m=50 0.4 0.35 Time Efficency (sec) 0.3 0.25 0.2 0.15 0.1 0.05 0 0 5 10 15 Increasing ||A|| 20 25 Pade Approximation Taylor Series RungeKutta45 PredictorCorrector

Efficency, RungeKutta23 stiff, A Cm m, m=50 2.5

2 Time Efficency (sec)

1.5

0.5

0 0

10 15 Increasing ||A||

20

25

(a)

(b)

Figure 10: Here we see that the least ecient is again the sti solver (black line in (b)). While the fastest is the Pad e approximation (red line in (a)).

19

x 10

Accuracy, A C

m m

, m=100
1.4 1.2 1 x 10
9

Accuracy, A Cm m, m=100 Taylor RungeKutta45

4 Relative Error

Pade RungeKutta23 stiff PredictorCorrector

Relative Error

0.8 0.6 0.4

1
0.2

0 0

10

20 30 40 Increasing ||A||

50

60

0 0

10

20 30 40 Increasing ||A||

50

60

(a)

(b)

Figure 11: Here we see similar results to gure 7. For positive denite 100 100 matrices we see that the accuracy is worst for predictor-corrector method (cyan line is (a)) and best for Taylor (blue line in (b)) and Pad e approximations (red line in (a)).

20

Efficency, A Cm m, m=100 0.7 0.6 Time Efficency (sec) 0.5 0.4 0.3 0.2 0.1 0 0 10 20 30 40 Increasing ||A|| 50 60 Pade Approximation Taylor Series RungeKutta45 PredictorCorrector

Efficency, RungeKutta23 stiff, A Cm m, m=100 20

Time Efficency (sec)

15

10

0 0

10

20 30 40 Increasing ||A||

50

60

(a)

(b)

Figure 12: Here we see that the least ecient is again the sti solver (black line in (b)) which takes 20 seconds. While the fastest is the Pad e approximation (red line in (a)) which takes under 0.1 seconds.

21