Multiagent Systems

Repeated Games

© Manfred Huber 2018 1

 Repeated Games
 Repeated games are an extension to normal
form games where the same single-stage
game is played multiple times sequentially
 What information is available to the agents ?
 Utility of a repeated game ?
 Strategies in repeated games ?
 There are two main types of repeated games
 Finitely-repeated games
 Infinitely-repeated games
© Manfred Huber 2018 2
 Finitely-Repeated Games
 A fixed single-stage game is repeated a finite
number of time
 After each game agents are aware what the other
agents did in the previous stages
 Payoff for the repeated game is the sum of the
payoffs in the stage games
 Finitely-repeated games can be translated
into imperfect information games in extensive
form
© Manfred Huber 2018 3
 Finitely-Repeated Games
 Twice repeated Prisoners’ Dilemma
C S C S
C -5, -5 -1, -10 C -5, -5 -1, -10
S -10, -1 -3, -3
 S -10, -1 -3, -3

C 1 S
2 2
C S C S

C 1 S C 1 S C 1 S C 1 S
2(-5,-5) 2 2(-1,-10)2 2(-10,-1)2 2(-3,-3) 2
C S C S C S C S C S C S C S C S

(-10, (-6, (-15, (-8, (-6, (-2, (-11, (-4, (-15,(-11, (-20, (-13, (-8, (-4, (-13, (-6,
-10) -15) -6) -8) -15) -20) -11) -13) -6) -11) -2) -4) -8) -13) -4) -6)
© Manfred Huber 2018 4
 Strategies in
Finitely-Repeated Games
 Strategy space is richer than the
strategy space for the single stage
normal form game
 Strategies can depend on actions played in
previous iterations of the game
 Stationary strategies are a special case where the
strategy reduces to the same strategy for each
single-stage game

© Manfred Huber 2018 5

 Nash Equilibria in
Finitely-Repeated Games
 Nash equilibria can be computed using the
corresponding extensive form game
 Exponential increase in complexity with the number of
repetitions
 A stationary strategy repeating a Nash
equilibrium for the stage game is a Nash
equilibrium for the repeated game
 In cases where the stage game has a dominant
strategy, backward induction can be used to
determine a Nash equilibrium
© Manfred Huber 2018 6
 Infinitely-Repeated Games
 A fixed single-stage game is repeated an
infinite number of times
 After each game agents are aware what the other
agents did in the previous stages
 Can not be converted into an extensive form game
 Conversion would require an infinite tree
 Payoff for the repeated game can not be
calculated as the sum of payoffs of the stage
games
 Sum of payoffs is in general infinite (not a usable utility)
© Manfred Huber 2018 7
 Utility/Payoff in
Infinitely-Repeated Games
 Average reward: game payoff is measured as
the average payoff of each stage game

 Future discounted reward: game payoff is

measured as the discounted sum of payoff of
the stage games
¥
u = å b j rj
j=1
© Manfred Huber 2018 8
 Strategies and Equilibria in
Infinitely-Repeated Games
 A pure strategy defines an action choice
for every stage game (decision point)
 Strategies are infinite
 There is an infinite number of pure
strategies
 Famous strategies:
 Tit-for-tat
 Trigger strategy
© Manfred Huber 2018 9
 Nash Equilibria in
Infinitely-Repeated Games
 A stationary strategy repeating a Nash equilibrium for
the stage game is again a Nash equilibrium for the
repeated game
 Other equilibria can depend on the choice of payoff
function and the discount parameter (in the case of
future discounted reward)
 There can be an infinite number of pure strategy
equilibria
 There is no way to construct the induced normal form
or the sequence form for infinitely repeated games
© Manfred Huber 2018 10
 Computing Nash Equilibria in
Infinitely-Repeated Games
 Nash equilibria for infinitely repeating games can not be
calculated by reducing the problem to a normal (or
extensive) form game
 The payoff received in a Nash equilibrium of an infinitely
repeating game can be characterized independently of
the strategy by looking only at the stage game
 The average payoffs in an equilibrium have to be achievable
under a mixed strategy in the stage game
 The average payoff achieved for player i in an equilibrium has
to be at least as big as the one received if the other players
adopt a minmax strategy against it
© Manfred Huber 2018 11
 Computing Nash Equilibria
The Folk Theorem
 Consider an n-player game G=(N,A,u) and a
payoff vector r=(r1, …, rn)
 The minmax value of an agent is the value it
can achieve if all other agents are antagonistic
(i.e. attempt to minimize the agent’s payoff)
 minmax value: v i = min max ui (si,s-i )
s- i ÎS- i si ÎSi

 minmax represents a conservative value below

which other agents can not push agent i ’s payoff
© Manfred Huber 2018 12
 Enforceability
 A payoff profile is enforceable if for every
agent the payoff is at least as high as its
minmax value

"i : ri ³ v i

© Manfred Huber 2018 13

 Feasibility
 A payoff profile is feasible if it is a convex,
rational (non-negative) combination of the
outcomes of the stage game

$a a Î Q+0 : åa a =1 Ù
a ÎA

"i ri = åa u (a)
a i
a ÎA

© Manfred Huber 2018 14

 The Folk Theorem
 In any n-player infinitely repeated game with stage
game G, the following holds:
 If r is the payoff profile for a Nash equilibrium in the infinitely
repeated game with average rewards, then ri is enforceable
 If ri were not enforceable then agent i would receive a better
payoff when pursuing the minmax strategy
 If r is feasible and enforceable then r is the payoff profile for
some Nash equilibrium of the infinitely repeated game with
average rewards
 If r is feasible and enforceable then we can construct a Nash
equilibrium strategy in the form of a trigger strategy that follows
the mixed strategy using to the rational weights for enforcability
and the minmax strategy after an agent diverts from the policy.
© Manfred Huber 2018 15
 The Folk Theorem
Constructing Equilibrium Strategies
 Find a payoff profile that is enforceable and feasible
 Find the rational coefficients for feasibility
æ ba ö
"i : ri = åç ÷ui (a) , b a Î Z +0 , g ÎN
a ÎAè g ø

 Construct a trigger strategy that performs the following:

 As long as no agent deviates from it, follow a pure strategy
that repeatedly cycles through all pure strategies a of the
stage game G, executing them βa times
 If in any cycle one of the agents deviates from this, the other
agents will follow that agent’s minmax strategy for the rest
of the game
© Manfred Huber 2018 16
 Infinitely-Repeated Games and
Bounded Rationality
 Solutions to infinitely repeated games assume:
 Arbitrarily deep reasoning
 Arbitrarily deep mutual modeling
 In real problems this is rarely feasible
 Discounted reward can limit the reliance on
perfect future information
 Bounded rationality can address this problem
 Approximateε-Nash equilibria
 Machine games: Games played by automata (limiting
memory and strategy complexity)
© Manfred Huber 2018 17