Reliability Engineering and System Safety 90 (2005) 25–29

www.elsevier.com/locate/ress

Optimization for condition-based maintenance

with semi-Markov decision process*
Dongyan Chena,*, Kishor S. Trivedib
a
Department of Computer Sciences and Computer Engineering, Xavier University of Louisiana, New Orleans, LA 70125, USA
b
Center for Advanced Computing and Communications, Duke University Durham, NC 27708-0294, USA

Received 1 July 2003; accepted 26 January 2004

Available online 25 December 2004

Abstract
The semi-Markov decision model is a powerful tool in analyzing sequential decision processes with random decision epochs. In this paper,
we have built the semi-Markov decision process (SMDP) for the maintenance policy optimization of condition-based preventive
maintenance problems, and have presented the approach for joint optimization of inspection rate and maintenance policy. Through numerical
examples, the improvement of this method is compared with the scheme, which optimizes only over the inspection rate. We also find that
under a special case when the deterioration rate at each failure stage is the same, the optimal policy obtained by SMDP algorithm is a
dynamic threshold-type scheme with threshold value depending on the inspection rate.
q 2004 Elsevier Ltd. All rights reserved.

Keywords: Preventive maintenance; Reliability; Availability; Markov decision process; Optimization

1. Introduction maintenance to recover the system to the previous stage of

Preventive maintenance is defined as the activity
undertaken regularly at pre-selected intervals while the
device is satisfactorily operating, to reduce or eliminate the
accumulated deterioration [5], while repair is the activity to
bring the device to a non-failed state after it has experienced
a failure. When the cost incurred by a device failure is larger
than the cost of preventive maintenance (this cost could be
cost of downtime, repair expenses, revenue lost, etc.), it is
worthwhile to carry out preventive maintenance.
Generally, there exist two types of preventive mainten-
ance schemes, i.e. condition based and time based preventive
maintenance [3]. For condition based preventive mainten-
ance, the action taken after each inspection is dependent on
the state of the system. It could be no action, or minimal
*
This research was supported in part by the Air Force Office of Scientific
Research under MURI Grant No. F49620-00-1-0327, and in part by
DARPA and US Army Research Office under Award No. C-DAAD19 01-1-
0646. Any opinions, findings, and conclusions or recommendations
expressed in this publication are those of the author(s) and does not
necessarily reflect the view of the sponsoring agencies.
* Corresponding author. Fax: C1 504 5207908.
E-mail address: cdongyan@xula.edu (D. Chen).
0951-8320/$ - see front matter q 2004 Elsevier Ltd. All rights reserved.
doi:10.1016/j.ress.2004.11.001
degradation, or major maintenance to bring the system to as
good as new state. For time based preventive maintenance,
the preventive maintenance is carried out at pre-determined
time intervals to bring the system to as good as new state [7].
In this paper we focus on the condition based preventive
maintenance.
Sim and Endrenyi introduced the multi-stage exponential
device failure model in [5], in which the idea of minimal
preventive maintenance was introduced. The minimal
preventive maintenance is defined as the preventive
maintenance activity with limited effort and effect [4]. For
deterioration failures modeled as several stages of expo-
nential distributions, minimal maintenance restores the
system to the previous deterioration stage. Corresponding
to the minimal preventive maintenance, the idea of major
maintenance is defined as the maintenance operation by
which the device is restored ‘as good as new’ status.
With condition based preventive maintenance, the
maintenance action taken after each inspection is dependent
on the state of the system. There could be no action, or
minimal maintenance to recover the system to the previous
failure stage, or major maintenance to bring the system to
as good as new state. Hosseini et al. [2] introduced
26 D. Chen, K.S. Trivedi / Reliability Engineering and System Safety 90 (2005) 25–29

the threshold-type policy for the maintenance action. No g minimal preventive maintenance threshold
maintenance action is taken if the device deterioration stage is b major preventive maintenance threshold
found to be smaller than the minimal maintenance threshold; li failure rate at stage i
minimal maintenance is performed if the device deterioration Fin time to inspection trigger interval distribution
stage is found to be between the minimal maintenance Fd distribution of the time to carry out an inspection
threshold and the major maintenance threshold; and major Fm preventive minimal maintenance duration distribution
maintenance is carried out if the device deterioration stage is FM preventive major maintenance duration distribution
larger than the major maintenance threshold. This model was FR failure repair time distribution
captured by a stochastic Petri net, and its optimal inspection F system failure state
interval to maximize the system availability is presented.
Closed-form results for such threshold-type condition-based Consider the epoch at which the system fails and enter
maintenance problems are reported in [1]. state F as the fictitious decision epoch. The only action at
Above works have only considered the optimization of this epoch is to repair the system to as good as new state. Let
system parameters. We notice that another important factor in i2IZ{0,1,.,kC1} define a system state where the system
determining the overall system availability is the maintenance is found to be in deterioration stage i with 0%i%k, or failure
policy, i.e. the action to be taken at each deterioration stage. state F represented by iZkC1, during an inspection.1 For
Improperly designed policies may considerably hamper the simplification of representation, in the following of our
ability of the system to meet certain design objectives, even paper we call state 0%i%k inspection state and state iZkC
with carefully chosen parameters. For this reason, in this paper 1 failure state.
we aim at joint optimization of both system parameters and In each state, a decision needs to be made on the action to
system maintenance policy. perform according to the system deterioration stage. The
One possible approach to tackle this problem is by solving action should include both the parameter(s) to determine the
the system model and search for the optimal combination next inspection time q, and the maintenance action (do
exhaustively. However, when the number of failure stages is nothing, perform minimal maintenance, or perform major
large, this work is cumbersome. For this reason, we maintenance). We represent the action at the nth decision
formulate the semi-Markov decision process (SMDP) epoch as a two-tuple (An, q) where An is the maintenance
model [6] for the condition-based maintenance problem, action to be performed and q is the mean time to next
based on the assumption that the system behavior may be inspection. Then, for states iZ0,.,k the possible action
captured by Markovian models. Besides deterioration fail- An is
ures, in our model, we have also considered Poisson failures,
8
which are defined as the type of failures under which the > 0; no action is taken
<
system fails abruptly rather than gradually, as with the
An Z 1; minimal maintenance is performed (1)
deterioration failures. For Poisson failures, we assume >
:
the repair action is to restore the system to the operable 2; major maintenance is performed
state it was just in before the failure.
The joint optimization of system parameters (the inspec- and in state iZkC1 the only action is to repair the system to
tion rate in our case) and the maintenance policy is as good as new state with q as the next inspection time
performed by taking the inspection rate as input parameter distribution parameter.
to the SMDP model. For each individual inspection rate the Let Yn, n2NZ{0,1,.} be the system state at the nth
SMDP model is solved for the optimal policy, based on decision epoch. Then, the state transition probability could
which the system CTMC model is constructed and solved. be derived.
From these results, we may thus obtain the best combinations For the transition probability from one inspection state
of system inspection rate and system maintenance policy. with deterioration stage i to the next inspection state with
This paper is organized as follows: In Section 1 we deterioration stage j we have
introduce the notation, and the Markov model and the SPN
model for the preventive maintenance problem. In Section 2 PðYnC1 Z jjYn Z i; An Z a; qn Z qÞ
we formulate the MDP problem for determining the optimal 8 ðN
>
> j
maintenance policy, and in Section 3 numerical results are > P~ i ðtÞ dFin ðt; qÞ
> a Z0
>
>
presented. Finally, Section 4 concludes the paper. < ðN0
j
Z P~ iK1 ðtÞ dFin ðt; qÞ a Z 1 (2)
>
> ðN
0
2. Notation and the SMDP formulation >
>
>
> j
: P~ 0 ðtÞ dFin ðt; qÞ a Z2
0
In this paper, the following notations are used:

k total number of deterioration stages 1

We assume that the failure state F can be discovered without any
i running index deterioration stage inspection.
 D. Chen, K.S. Trivedi / Reliability Engineering and System Safety 90 (2005) 25–29 27

j
where 0%i%kC1, 0%j%k, and P~ i ðtÞ is the probability that and find the stationary policy R(n) that minimize the
the system state changes from i to j without any inspection above equation.
j
event. Computation of P~ i ðtÞ is shown in Appendix A. Step 2. Compute the bounds mn Z minx2I ðVn ðxÞK VnK1 ðxÞÞ
The transition probability from an inspection/failure state and Mn Z maxx2I ðVn ðxÞK VnK1 ðxÞÞ:
to failure state F is If 0%(MnKmn)%3mn, where 3 is a pre-specified
PðYnC1 Z k C 1jYn Z i; An Z a; qn Z qÞ accuracy number, the algorithm is stopped with the
policy R(n). Otherwise, go to step 3.
ðN
kC1 Step 3. nZnC1 and go to step 1.
Z ð1 K Fin ðt; qÞÞdðP~ i ðtÞÞ (3)
0

where 0%i%k.
The expected time to the occurrence of next decision
epoch, given current state i and action a is
8 ðN ðN
>
> ~ kC1
>
> ð1 K P i ðtÞÞð1 K F in ðtÞÞdt C Fd ðtÞ dt 0% i% k; a Z 0
>
> ðN
0 ð0N
>
>
>
> kC1
< ð1 K P~ iK1 ðtÞÞð1 K Fin ðtÞÞ dt C ðFd ðtÞ C Fm ðtÞÞ dt 0% i% k; a Z 1
tði; aÞ Z ð0
N ðN0
(4)
>
> ~ kC1
>
> ð1 K P0 ðtÞÞð1 K Fin ðtÞÞ dt C ðFd ðtÞ C FM ðtÞÞ dt 0% i% k; a Z 2
>
> ðN
0 ðN0
>
>
>
> ~ kC1
: ð1 K P0 ðtÞÞð1 K Fin ðtÞÞ dt C FR ðtÞ dt i Z k C1
0 0

The system operational cost is assigned as follows 3. Numerical results

Cost Meaning In the numerical evaluation of the MDP scheme, we

cm cost per unit time of downtime due to maintenance assume the time to inspection, inspection time, minimal
cd cost per unit time of downtime due to inspection maintenance time, major maintenance time and repair time
cR cost per unit time of downtime due to repair are all deterministically distributed. Then, we have
c0m cost for each minimal maintenance
c0M cost for each major maintenance Fin ðt; qÞ Z Uðt K qÞ; Fd ðtÞ Z Uðt K td Þ;
c0R cost for each repair
Fm ðtÞ Z Uðt K tm Þ; FM ðtÞ Z Uðt K tM Þ
Then, the cost function in each state x with action a can
be written as FR ðtÞ Z Uðt K tR Þ
8 ðN
>
> cd ð1 K Fd ðtÞÞ dt i Z0
>
>
>
>
> ðN
>
0
>
>
>
> c d ð1 K Fd ðtÞÞ dt 1% i% k; a Z 0
>
> ð0 ðN
< N
cði; aÞ Z cd ð1 K Fd ðtÞÞ dt C cm ð1 K Fm ðtÞÞdt C cm0 ; 1% i% k; a Z 1 (5)
>
> ðN
0 ð0N
>
>
>
> 0
>
> cd ð1 K Fd ðtÞÞ dt C cM ð1 K FM ðtÞÞ dt C cM ; 1% i% k; a Z 2
>
> ð0 0
>
> N
>
>
: cR ð1 K FR ðtÞÞ dt C cR0 : i Z k C1
0

The value iteration algorithm is summarized as follows: and the parameters are chosen as follows

Step 0. Choose V0(x) such that 0%V0(x)%mina k Z 10; td Z 0:5; tm Z 0:5; tM Z 0:5; tR Z 100:
(c(x,a)/t(x,a)) for all x. Let nZ1. and we assume the deterioration rate li at each stage is the
Step 1. Compute the function Vn(x), x2I, from same with a value of 0.03.
" The cost for each state is chosen in such a way that the
cðx; aÞ t X total cost represents the system steady-state availability, i.e.
Vn ðxÞ Z min ; C P V ðyÞ
a2AðxÞ;q tðx; aÞ tðx; aÞ y2I xay nK1 cdZ1, cmZ1 with all other costs equal to zero.

Fig. 1 shows a comparison between the MDP scheme and
t the scheme with fixed minimal and major maintenance
C 1K V ðxÞ; x 2I; ð6Þ
tðx; aÞ nK1 threshold. With MDP scheme, maximal availability of 0.962
28 D. Chen, K.S. Trivedi / Reliability Engineering and System Safety 90 (2005) 25–29

It may be observed that with more frequent inspections

(higher inspection rates), the major maintenance threshold
becomes larger, which means more minimal maintenance is
performed. Intuitively, this phenomenon may be explained
in such way that more frequent inspections lead to more
timely warning if the system is near to failure. To reduce
system operation cost or increase availability, it is thus
better to take risk to perform minimal maintenance even if
the system is in its higher deterioration stages.
While the threshold type policies obtained by SMDP
approach are easier to understand and employ, it is
worthwhile to point out that the optimal policy is not
necessarily threshold type in other situations, e.g. when the
deterioration rate are not the same or when the objective of
optimization is different from the steady-state availability.
In these situations, the optimal policy may be more
complicated than the threshold-type policies.
Fig. 1. Comparison between the threshold policy and the MDP policy.

may be achieved by choosing linZ0.021 while with fixed

threshold the maximal availability of 0.955 may be achieved 4. Conclusion
with linZ0.032. This figure demonstrates the improvement
achievable by the joint optimization approach over the In this paper, we have presented the application of MDP
optimization of only the inspection rate. algorithm in searching for the optimal maintenance policy
A deeper look into the optimal maintenance policies for condition based maintenance, and we have also
reveals that the policy obtained by MDP algorithm is also presented a joint optimization of inspection rate and its
threshold-type policy, but with different thresholds b and g corresponding maintenance policy. Under a special case
for different inspection rate lin For example, the optimal when the optimization objective is steady-state availability
inspection rate linz0002 according to Fig. 1. If we denote and the deterioration rate at each failure stage is the same,
the minimal maintenance threshold by g, and denote the we find that the optimal policy is a threshold-type
major maintenance threshold by b, the optimal maintenance maintenance policy.
policy corresponding to linZ0002 is gZ0 for minimal
maintenance threshold and bZ4 for major maintenance
threshold.
j
To further explore the relationship between the inspec- Appendix A. Calculation of P~ i ðtÞ
tion rate and maintenance policy, we have varied lin to
obtain the corresponding thresholds. Fig. 2 shows the result. For state 0%i%k, 0%j%kC1, the system balance
equation is
j
dP~ i ðtÞ
Z 0; j! i (A1)
dt
i
dP~ i ðtÞ i
Z Kli P~ i ðtÞ; (A2)
dt
iC1
dP~ i ðtÞ iC1 i
Z KliC1 P~ i ðtÞ C li P~ i ðtÞ; (A3)
dt

« (A4)

j
dP~ i ðtÞ j jK1
Z Klj P~ i ðtÞ C ljK1 P~ i ðtÞ; i! j% k (A5)
dt

kC1
X
k
m
P~ i ðtÞ Z 1 K P~ i ðtÞ (A6)
mZi
Fig. 2. Thresholds for the optimal maintenance policy.
 D. Chen, K.S. Trivedi / Reliability Engineering and System Safety 90 (2005) 25–29 29

and for state iZkC1, 0%j%kC1, we have X

k
kC1 j
0
P~ i ðsÞ Z 1 K P~ i ðsÞ: (A14)
dP~ kC1 ðtÞ 0 jZi
Z Kl0 P~ kC1 ðtÞ; (A7)
dt Q
with jmZi Z1 for j!i and the time domain results may be
j
dP~ kC1 ðtÞ obtained by either analytical or numerical inversion of
j jK1
Z Klj P~ kC1 ðtÞ C ljK1 P~ kC1 ðtÞ; 1% j% k (A8) (A13) and (A14).
dt

kC1
X
k
m
P~ kC1 ðtÞ Z 1 K P~ kC1 ðtÞ (A9)
mZ0
References
j j
and thus P~ kC1 ðtÞZ P~ 0 ðtÞ; 0% j% kC 1:.
Performing Laplace transform, for 0%i%k we have [1] Chen D-Y, Trivedi KS. Closed-form analytical results for condition-
based maintenance. Reliab Eng Syst Saf 2002;(76):43–51.
i i
sP~ i ðsÞ Z 1 K li P~ i ðsÞ (A10) [2] Hosseini MM, Kerr RM, Randall RB. An inspection model with
minimal and major maintenance for a system with deterioration and
j j jK1 Poisson failures. IEEE Trans Reliab 2000;49(1):88–98.
sP~ i ðsÞ Z Klj P~ i ðsÞ C ljK1 P~ i ðsÞ; i! j% k (A11) [3] Legat V, Zaludova AH, Cervenka V, Jurca V. Contribution to
optimization of preventive maintenance. Reliab Eng Syst Saf 1996;
kC1
X
k
m
51:259–66.
P~ i ðsÞ Z 1 K P~ i ðsÞ: (A12) [4] Sheu S-H, Yeh RH, Lin Y-B, Juang M-G. A Bayesian approach to an
mZi adaptive preventive maintenance model. Reliab Eng Syst Saf 2001;71:
j j 33–44.
and P~ kC1 ðsÞZ P~ 0 ðsÞ; 0% j% kC 1. [5] Sim SH, Endrenyi J. Optimal preventive maintenance with repair.
i0
Therefore, the Laplace transform of P~ i ðtÞ may be written IEEE Trans Reliab 1988;37(1):92–6.
as [6] Tijms HC. Stochastic modelling and analysis: a computational
approach. New York: Wiley; 1986.
QjK1
j l [7] Vaurio JK. On time-dependent availability and maintenance optimiz-
P~ i ðsÞ Z Qj mZi m ; i% j% k (A13) ation of standby units under various maintenance policies. Reliab Eng
mZi ðs C lm Þ Syst Saf 1997;56:79–89.