MATLAB WWW.MATLAB.NET.

CN

MATLAB WWW.MATLAB.NET.CN
1
1.1
dynamic programming
20 50 R. E. Bellman (multistep
decision process)principle of optimality

1957 Dynamic Programming

A G

2
1
3 6
2324

0.5

1.2
discrete-time
decision processcontinuous-time decision process
deterministic decision process
stochastic decision process
-35-


2
2.1

2.1.1
(step)
k = 1,2,, n 1
A k = 1 Bi (i =1,2) k = 2 Fi (i =1,2)
k = 6 n = 6 2 k =1,2,3,4

2.1.2
state

state variable
(set of admissible states) xk k
X k k 1 x2 B1 , B2 Bi
i (i =1,2) x 2 = 1 2 X 2 = {1,2}

n +1 x n +1 xn 1
x 7 G 1 x 7 = 1

2.1.3

decisioncontrol
decision variable
set of admissible decisions u k ( x k ) k xk
xk U k ( x k ) xk 1 u 2 ( B1 )

C1 , C 2 C 3 u 2 (1) = 1,2,3 U 2 (1) = {1,2,3}

2.1.4
policy x1
p1n ( x1 )

p1n ( x1 ) ={u1 ( x1 ), u2 ( x 2 ), , u n ( x n )} .
k xk p kn ( x k )
pkn ( x k ) = {u k ( x k ), , u n ( x n )} k = 1,2, , n 1 .
k j
p kj ( x k ) ={u k ( x k ), , u j ( x j )} .
(set of admissible policies)
P1n ( x1 ), Pkn ( x k ), Pkj ( x k )
2.1.5.

-36-

MATLAB WWW.MATLAB.NET.CN
equation of state transition
x k +1 = Tk ( x k , u k ), k =1,2,, n.
1
1 x k +1 = u k ( x k )
2.1.6.
(objective function)
Vkn ( x k , uk , x k +1 ,, x n +1 ) k = 1,2,, n
Vkn x k , u k ,Vk +1 n

Vkn ( x k , u k , x k +1 , , x n +1 ) =k ( x k , u k , Vk +1n ( x k +1 , u k +1 , x k +2 , x n +1 ))
k Vk +1 n
j x j u j v j ( x j , u j )
v j ( j =1,2, , n )

Vkn ( x k , u k , x k +1 ,, x n +1 ) = v j ( x j , u j )
j =k

Vkn ( x k , u k , x k +1 ,, x n +1 ) = v j ( x j , u j )
j =k

Vkn ( x k , uk , x k +1 ,, x n +1 ) = max (min) v j ( x j , u j ) .

k j n

k j Vkj ( x k , u k , x k +1 , x j +1 )
Vkn xk p kn

Vkn ( x k , p kn ) xk Vkn p kn optimal

value function f k ( x k )

f k ( xk ) =

opt

p kn Pkn ( x k )

Vkn ( x k , p kn )

opt max min

2.1.7
Vkn k
*
*
p kn
= {u k* ,, u n* } p1n
optimal policy
*
x1 ( = x1* ) p1n

{x1* , x 2* ,, x n* +1 } optimal trajectory

2.1.8

-37-

 fn+ 1(xn+ 1) = 01
f (x ) = o {vp(x ,ut) f (x ) k }= n,,1
kk
k k k k+ 1 k+ 1
uk Uk(xk)

f n +1 ( x k +1 ) = 0
f n +1 ( x k +1 ) =1
12
k = n + 1 k = 1

x k = Tk +1 ( x k +1 , u k +1 ), k = 1, , n

f1(x1= 01
f (x ) = o {pv (x t,u ) f (x ) k =}1,,n,
k+ 1 k+ 1
k+ 1 k+ 1 k+ 1 k k
uk+1 Uk+1(xk+1)

i
ii xk
X k
iii u k U k ( x k )
iv
v v k ( x k , u k ) Vkn

vi
-38-

MATLAB WWW.MATLAB.NET.CN
3

f n +1 ( x n +1 i ) =( x n +1 i ), i =1,2,, n n +1
(3)
X 1 = {x1 } = {x1* }
xk uk
xk
X k = {x ki | i = 1,2, , nk },

k = 1,2, , n .

u ki ( x ki )

U ki = {u ki( j ) | j = 1,2,, nki }, i = 1,2,, n k , k = 1,2,, n .

( j)
xk xki uki uki
( j)
( j)
Tk = Tk ( x ki , u ki ) v k = v ( x ki , u ki ) xk xki

f k( j ) ( x ki ) = v k ( x ki , u ki( j ) ) + f k +1 (Tk ( x ki , u ki( j ) )),

f k ( x ki ) = opt f k( j ) ( x ki ),

j = 1,2,, n ki , i = 1,2,, n k , k = n,,2,1.

34 f 1 ( x1* ) xki
*

u ki ( x ki ) x1* u ki ( x ki ) xk
*

u k ( x k ) {u1 ( x1 ), u 2 ( x 2 ), , u n ( x n )}

-39-

 f 1 ( x1* )
*
f k ( x ki ) u k ( x ki )

f k ( x ki ) f k 1 f k 1 f k 1 f k

u k* ( x ki ) u k* ( x k* )

{u1* ( x1* ), u 2* ( x 2* ), , u n* ( x n* )} {x1* , x 2* ,, x n* }

4

u1 , u 2 , , u n V1n ( x1 , u1u2 ,, un )

max

g
k =1

-40-

(u k )

MATLAB WWW.MATLAB.NET.CN
n

s.t.

u
k =1

= a, uk 0 .

g k (u k )
u k n x1 , x 2 ,, x n
u1 , u 2 , , u n

x1 = a , x k +1 = x k u k , k = 1,2,, n.
g k (u k ) x n +1 = 0

f k ( x k ) = max [ g k ( x k ) + f k +1 ( x k +1 )]
0 uk x k

0 x k a, k = n, n 1,,2,1
f n +1 (0) = 0 .
*
xk u k ( x k ) f 1 ( a )
*

{x k } {uk ( x k )}

ii

iii

iicurse of dimensionality
m n xk m n
f k ( x k ) n n = 3

5
5.1
1 shortest Path Problem
x k +1 = u k ( x k )
d k ( x k , u k ( x k ))
-41-

 f k ( x k ) xk

f k ( x k ) = min [d k ( x k , u k ( x k )) + f k +1 ( x k +1 )], k = n,,1;

uk ( x k )

f n +1 ( x n +1 ) = 0.
l AB 1C 2 D1 E 2 F2 G
18
5.2
2 Production planning problem
xk uk
d k

x k +1 = x k + u k d k , x k 0, k = 1,2,, n.
(5)
a
b
c

a + b ku, uk > 0
vk ( xk , uk ) = c kx+
0

(6)

Vkn vk f k ( x k ) k xk

f k ( x k ) = min [v k ( x k , u k ) + f k +1 ( x k +1 )], k = n,,1.

u k U k

U k

xn0 +1
f n +1 ( x n0+1 ) = 0.

5~7
5.3

resource allocating Problem

4 u g (u )
h (u ) a1 b1

0 < b1 < a1 < 1 m n

n g ( u ) = u
h (u ) = u > > 0

k = 1,2, , n x k k u k
k x k u k

a b a = 1 a1 b = 1 b1
a < b k x k uk

x k +1 = au k + b( x k u k )
-42-

MATLAB WWW.MATLAB.NET.CN
v k k
vk ( x k , uk ) = g (uk ) + h( x k uk )

f k ( x k )

f k ( x k ) = max [v k ( x k , uk ) + f k +1 ( x k +1 )],
0uk xk

0 x k m,

k = n, ,2,1.

(10)

f n +1 ( x n +1 ) = 0,

0 x n +1 m.

11

v k g, h k
g (u ) = u h (u ) = u 10
[ v k ( x k , u k ) + f k +1 ( x k )] u k 0 uk x k
uk = 0 u k = x k

1. 1000 AB y A
5 y y B 4 y
A 20% B 10% 5
( Matlab )
2. 4

C
B
D
A

15
18
21
24

19
23
22
18

26
17
16
19

19
21
23
17

3. E1 , E 2 , E 3
8000

E3
E3
E1
E2
E1
E2

z =1
z=2

0.3
0.2
0.1
1
3
2
0.4
0.5
0.2
2
5
3
0.5
0.9
0.7
3
6
4
z =3

-43-