BACKTESTING GENERAL SPECTRAL RISK MEASURES

WITH APPLICATION TO EXPECTED SHORTFALL

NICK COSTANZINO AND MIKE CURRAN

Abstract. In this note, we present a simple, practical and easily implementable coverage test to backtest any spectral risk measure. Our
test gives a single decision at a specified confidence level and is perfectly
consistent with the binomial test for VaR. Particular attention is given
to the special case of Expected Shortfall.

Contents
1. Background and Motivation
2. VaR and Spectral Risk Measures
3. Deriving the Backtest Statistic and Coverage Test
4. Application to Expected Shortfall
5. Conclusions
Acknowledgements
References

1
3
4
9
10
10
11

1. Background and Motivation

Let
be a sequence of historical trading days and {Li }N
i=1 the corresponding realized trading losses. One way to assess the accuracy of the
calculation of VaR for those trading days is to backtest VaR using the following coverage test.
{ti }N
i=0

For each trading day i = 1, ..., N , let VaRi () denote the VaR at level
(i)
and XVaR () := 1{Li V aRi ()} {0, 1} denote the VaR failure indicator.
N () [0, 1] for level [0, 1] over N
We define the VaR failure rate XVaR
trading days as

Date: February 21, 2015.

1

N
XVaR
()

N
1 X (i)
:=
XVaR ()
N

(1.1)
=

1
N

i=1
N
X

1{Li VaRi ()} .

i=1

N
Hence XVaR
is simply the average number of VaR breaches at level over
N
N trading days. By appealing to the Central Limit Theorem XVaR
is approximately normal under the null-hypothesis with expected value and
N
variance (1 )/N . Thus XVaR
admits a Z-test with Z-score

(1.2)

N
ZVaR
()

b N ()
X
VaR
p
(1 )

b N is the realized/empirical VaR failure rate (1.1) over the N

where X
VaR
trading days. We can then define a one-sided or two-sided Z-test through
N ()) for one-sided and
the cumulative distribution . For instance, (ZVaR
N
(2|ZVaR ()|) for two-sided.
In this note we develop an extension of the VaR coverage backtest to
any Spectral Risk Measure, in particular to Expected Shortfall. Similar to
the VaR coverage test which serves as the basis for the Basel Committees
Traffic Light test, our coverage test does not test independence (for more on
VaR and backtesting methodologies c.f. [9, 16]). As we shall see in Section
3 the coverage test for Spectral Risk Measures essentially amounts to a joint
test of a continuum of weighted VaR quantiles and gives a single decision at
a fixed confidence level. The key to the method is to show that the Spectral
Measure Failure Rate defined by (3.2) is asymptotically normal under the
null hypothesis and therefore admits a formal Z-test as in Theorem 3.5.
The motivation for developing a coverage test for Expected Shortfall and
other Spectral Risk Measures is that ES is gaining traction as a risk measure
and could soon replace VaR as the workhorse for Risk Management. This is
borne from Regulators desire to better capture tail risk, since by definition
VaR does not provide any insight into the potential losses in the event of
a loss in excess of the VaR level. For instance its Fundamental Review of
the Trading Book [13], the Basel Committee proposes to replace VaR with
ES which captures tail risk better. In addition, since ES is a coherent risk
measure [3], it allows for more straightforward allocation of capital to the
underlying books of exposure and hence reasonable to use for determining
standardized capital requirements.
2

Despite the potentially prominent role of Expected Shortfall in Risk Management, accepted methods to backtest it are still elusive. Worse, there
have been growing concerns that ES is in fact not even backtestable and
thus should not be adopted as a risk measure. The claims that ES is not
backtestable range from mainstream articles [10, 11] to research articles [12].
The claim can be traced back to some observations that ES does not possess
some mathematical properties such as elicitability [14]. While it is true that
if a risk measure has elicitable structure there is particular way to backtest it
via the scoring function, lack of elicitability need not imply lack of backtestability. For example, as noted in [7], backtesting methods for VaR that are
used in practice do not use any structure from elicitability. In fact, recently
Acerbi & Szekely [2] make a strong argument that elicitability has nothing
to do with backtesting at all, but rather only model selection. Hence, the
mathematical property of elicitability may not be as important as thought
in the practical implementation of an backtesting algorithm.
In the midst of claims that ES is not backtestable, some backtesting
approaches have nonetheless been proposed. These include the censored
Gaussian approach introduced by Berkowitz [8], the functional delta approach used by Kerhoff and Melenberg [17] and the saddle-point technique
introduced by Wong [18] and later extended by Graham and Pal [15]. As
in every statistical method, each of these different approaches have their
strengths and weaknesses, and our method should be seen as complementary to these. In particular, our test gives a single decision at a specified
confidence level.

2. VaR and Spectral Risk Measures

Before deriving our coverage test in Section 3, we review the basic definitions of VaR and Spectral Risk Measures for completeness.
Definition 2.1. (Value-at-Risk) Suppose X is a random variable with cumulative distribution function FX . The Value-at-Risk (VaR) of a random
variable X with confidence level (0, 1) is given by
(2.1)

VaR() := inf{z R : FX (z) }.

VaR has been criticized as a risk measure in that it is not subbadditive

and does not take into account the severity of losses beyond level . In [4, 5]
Artzner, Delbaen, Eber, and Heath outline some basic properties which any
good risk measure should posses and call these measures Coherent. One
class of such measures are Spectral Risk Measures, which can be thought of
weighing VaR by a spectrum having particular properties.
3

Definition 2.2. (Admissible Risk Spectrum) We say L1 ([0, 1]) is an

admissible risk spectrum if
i. is non-negative
ii. is non-increasing
iii. kk1 = 1.
Definition 2.3. (Spectral Risk Measure) Suppose X is a random variable
with cumulative distribution function FX and is an admissible risk spectrum. Then we say that M defined by
1

Z
(2.2)

M :=

(p)VaR(p)dp
0

is a Spectral Risk Measure with risk spectrum .

Note that M depends on the distribution X through VaR (2.1) even
though we suppressed this dependence in our notation.
Remark 2.4. If (p) = Dirac (p), then MDirac = VaR(). However,
Dirac (p) violates both Properties ii and iii in 2.3 and thus not an admissible
risk spectrum. This explains why VaR is not a spectral risk measure.
In [1], C. Acerbi proved that Spectral Risk Measures (2.2) are Coherent.
These properties have made Spectral Risk Measures, and ES in particular,
a favourite of Regulators. For instance it is the measure that is proposed
to replace VaR under Basel III. For ES to be widely adopted, reasonable
backtesting methods must be developed. We now construct the coverage
backtest.

3. Deriving the Backtest Statistic and Coverage Test

N
In analogy with the VaR failure rate XVaR
described in the Introduction,
we define a Spectral Risk Measure failure rate XN which will serve as our
test statistic for our coverage test for Spectral Risk Measures.

Definition 3.1. (Spectral Risk Measure Failure Rate) For an admissible

(i)
spectrum , let XSR [0, 1] be defined by

(3.1)

(i)
XSR ()

Z
=
0

(p)1{Li VaRi (p)} dp.

N [0, 1] for admissible

We define the Spectral Risk Measure failure rate XSR
risk spectrum as
4

N
1 X (i)
XSR ()
N
i=1
N Z 1
X
1
(p)1{Li VaRi (p)} dp.
=
N
0

N
XSR
() :=

(3.2)

i=1

Remark 3.2. In contrast to VaR where failure is a discrete event with value
(i)
either zero or one (i.e. XVaR := 1{Li VaRi ()} {0, 1}) the corresponding
failure for Spectral Measures is a continuous variable with value between
R1
(i)
zero and one (i.e. XSR := 0 (p)1{Li VaRi ()} [0, 1]) and depends on the
severity of the failure.
To understand this better, we may alternatively write the Spectral Risk
Measure Failure Rate as
N
XSR
()

(3.3)

N Z
1 X 1
:=
(p)1{Li VaRi (p)} dp
N
i=1 0
N Z
1 X 1
=
(p)1{VaR1 (Li )p} dp
i
N
0
i=1
N Z
1 X 1
=
(p) dp
N
VaR1
i (Li )
i=1

N
1 X
(VaR1
=1
i (Li ))
N
i=1

where

= .

Definition 3.3. (Null Hypothesis for Coverage Test) The null-hypothesis

for the Spectral Risk Measure Coverage Test is
(3.4)

(i)

H0 : {X }N
i=1 i 6= j, and P [Li VaRi (p)] = p p supp
(i)

The first hypothesis essentially means that {X }N

i=1 are i.i.d. Since the
Alternate Hypothesis is complement of the Null, our coverage test cannot
differentiate between rejection due to the i.i.d. hypothesis or rejection due
to the distribution hypothesis. This is a typical shortcoming of classical
coverage tests, including the standard VaR coverage test.
Proposition 3.4. (Mean and Variance of XN under h0 ) Consider the random variable XN [0, 1] defined by (3.2). Then under the null-hypothesis,
(3.5)

N
:= E[XSR
()] =

(p) p dp
0

and

(3.6)
1
N
2 := V[XSR
()] =
N

1Z p

(p)(q)qdpdq

2
0

2 !

Z

(p)pdp

(i)

Proof. To prove (3.5) we first consider a single trading day failure XSR and
compute

(i)
E[X ()]

Z
=E
0
1

Z
=
0

(3.7)

(p)1{Li VaRi (p)} dp

(p)E[1{Li VaRi (p)} ]dp

(p) P[Li VaRi (p)]dp

=
0
1

Z
=

(p) p dp
0

Thus

N
1 X (i)
N
XSR ()
E[XSR ()] := E
N

"

i=1

=
(3.8)

1
N

N
X

h
i
(i)
E XSR ()

i=1

N Z
1 X 1
=
(p) p dp
N
i=1 0
Z 1
=
(p) p dp
0

Hence the expected value of the average over N trading days is equal to the
expected value of a single trading day.
(i)

To prove (3.6) we again first consider a single trading day failure XSR and
compute
6

(3.9)
(i)
E[(XSR ())2 ]

"Z

=E
0

Z

=E
0

(p)1{Li VaRi (p)} dp

(p)1{Li VaRi (p)} dp

1Z p

Z

(q)1{Li VaRi (q)} dq

(p)(q)1{Li VaRi (p)} 1{Li VaRi (q)} dp dq

= 2E
0

2 #

1Z p

Z

(p)(q)1{{Li VaRi (p)}{Li VaRi (q)}} dp dq

Z 1 Z p

= 2E
(p)(q)1{{Li VaRi (q)} dp dq assuming q p
= 2E

0
0
1Z p

(p)(q)P[Li VaRi (q)]dp dq

=2
0

0
1Z p

(p)(q) q dp dq

=2
0

Using (3.5) we then have

(i)

(i)

(i)

V[XSR ()] := E[(XSR ())2 ] E[XSR ()]2

Z
Z 1Z p
(p)(q)qdpdq
=2

(3.10)

(p)pdp

2

Finally,
#
N
X
1
(i)
N
V[XSR
()] = V
XSR ()
N
i=1

N
X
X
1

(i)
(i)
(j)
V[XSR ()] +
corrhXSR (), XSR ()i
= 2
N
|
{z
}
i=1
"

(3.11)

i6=j

1
= 2
N
1
=
N

N
X

Z
(p)(q)qdpdq

2
0

i=1

= 0 under H0 (3.4)

1Z p

Z
(p)(q)q dp dq

2
0

(p)pdp

1Z p
0

2 !

2 !

(p)pdp


N admits a Z-test) The Spectral Measure Failure Rate X
Lemma 3.5. (XSR

is asymptotically normal under the null hypothesis and therefore admits a

Z-test.
7

(i)

Proof. Under the null hypothesis, the sequence {XSR }N

i=1 is i.i.d. with
2
bounded mean and variance given by (3.5) and (3.6) respectively.
Thus by the Lindeberg-Levy Central Limit Theorem

(3.12)

D
N
N XSR
() N (0, 2 ).
N

In fact, the convergence is pointwise and uniform, so that

(3.13)

lim kP

N +

i
N
N (XSR
) (/ )kL
= 0.

N is approximately normal and therefore

Therefore, for large enough N , XSR
admits a Z-test.


As usual, it would be interesting to get precise error bounds on the difN for finite N and its limiting distriference between the distribution of XSR
bution. This would allows us to control the error in some measure and let
us choose N to within some tolerance. For instance, since the third moment
N is uniformly bounded we can appeal to the Berry-Esseen theorem to
of XSR
N and the
give point-wise bounds on the difference between the CDF for XSR
limiting CDF. However, we do not pursue that direction here. We finally
arrive at our main Theorem.
Theorem 3.6. (Coverage Test for Spectral Risk Measures) Let and
be the mean and standard deviation of the Spectral Measure Failure Rate
N (3.2) under the null hypothesis given by
XSR
Z
(3.14)

(p) p dp
s
Z 1
2
Z 1Z p
1
=
2
(p)(q)q dp dq
(p)pdp
N
0
0
0
0

(3.15)

N defined by
Then the Z-score ZSR

(3.16)

N
ZSR
() :=

b N ()
X
SR

N and a Coverage Test for M .

defines a Z-test for XSR

N admits a Z-test with standard

Proof. By Lemma 3.5 the test statistic XSR
N under the null
Z-score. By Proposition 3.4 the mean and variance of XSR
hypothesis are given by (3.5) and (3.6) respectively leading to (3.16).

8

4. Application to Expected Shortfall

Expected Shortfall is a particular Spectral Risk Measure introduced in [4,
5] which has gained popularity since being introduced as a possible measure
to replace VaR. It is given by the average VaR above a threshold ,
(4.1)

1
ES() :=

VaR(p)dp.
0

In order to use the Coverage Test in Theorem 3.6 to backtest ES, we need to
write (4.1) as a Spectral Risk Measure with a specific choice of risk spectrum
ES .
Definition 4.1. (Expected Shortfall) The Expected Shortfall is a special
case of a Spectral Risk Measure with risk spectrum given by
1
1
.
{0p}
Thus, the ES risk measure (4.1) can be written as

(4.2)

ES (p) :=

Z
(4.3)

MES () =
0

VaR(p)
dp.

Since ES is an admissible risk spectrum, MES enjoys all the mathematical properties of Spectral Risk Measures.
To derive the coverage test for ES, we recall Definition 3.1 and define the
N as
Expected Shortfall Failure Rate XES

(4.4)

N
XES
()

Z
N
1 X 1
:=
1
dp
N
0 {Li VaRi (p)}
i=1

N is
which is simply (3.2) with = ES (4.2). Hence by Theorem 3.6, XES
asymptotically normal and admits a Z-test. To calculate the Z-score we
N under the null-hypothesis.
need to calculate the mean and variance of XES
To do this we substitute the Expected Shortfall risk spectrum ES (4.2) into
(3.5) and (3.6) and obtain
1

Z
ES () =

ES (p)p dp
0

(4.5)

=
2

and
9

p dp
0

2
ES
()

=
=

(4.6)
=
=


 Z 1Z p
1
2
ES (p)ES (q)q dp dq ES
2
N
 0Z 0Z p

1
2
2
q dp dq ES
N 2 0 0
 Z

1
2
1
1 2
p dp
N 2 0 2
4


4 3
N
12

2 we can use Theorem 3.6 to compute the Z-score which

Given ES and ES
forms basis for our Coverage Test. For completeness we write this as a
Corollary to Theorem 3.6.

Corollary 4.2. (Coverage Test for Expected Shortfall) The Expected Shortfall measure MES (4.1) admits a Z-test with Z-score
b N () ES ()
X
ES
ES ()
!
bES ()

2X
.
= 3N p
(4 3)

N
ZES
() =

(4.7)

This is clearly a test of the -tail in quantile space rather than dollar
space. We define a mapping between the two spaces in a subsequent paper.
5. Conclusions
In this note we presented a simple coverage test for any Spectral Risk
Measure, including Expected Shortfall. The test gives a single decision at
a specified confidence level and is complementary to other testing methods,
most notably the saddle-point techniques in [15, 18]. It would be interesting to compare the backtesting performance of our coverage test with the
methods mentioned in the Introduction.
Acknowledgements
The authors would like to thank Carlo Acerbi (MSCI) and Janos Pal
(BMO) for insightful discussions on backtesting risk measures, as well as
the anonymous referee for a critical reading of the manuscript.

10

References
[1] C. Acerbi, Spectral measures of risk: A coherent representation of subjective risk
aversion, Journal of Banking and Finance, 26, (2002), 1505-1518.
[2] C. Acerbi & B. Szekely, Backtesting Expected Shortfall, to appear in Risk Magazine, 2014.
[3] C. Acerbi & D. Tasche, On the coherence of Expected Shortfall, Journal of Banking
and Finance, Volume 26, Issue 7, July 2002, Pages 1487-1503.
[4] P. Artzner, F. Delbaen, J.M. Eber, & D. Heath, Thinking Coherently, Risk,
Vol 10, No. 11, (1997) 68-71.
[5] P. Artzner, F. Delbaen, J.M. Eber, & D. Heath, Coherent Measures of Risk,
Mathematical Finance, Vol. 9, Issue 3, (1999) 203-228.
[6] Basle Committee on Banking Supervision, Supervisory Framework for the use
of Backtesting in Conjunction with the Internal Models Approach to Market Risk
Capital Requirements, January 1996.
[7] F. Bellini & V. Bignozzi, Elicitable Risk Measures, Preprint, December 2013.
[8] J. Berkowitz, Testing Density Forecasts, with Applications to Risk Management,
Journal of Business and Economic Statistics, Vol 19, No 4, (2001) 465-474.
[9] S.D. Campbell, A Review of Backtesting and Backtesting Procedures, Finance and
Economics Discussion Series, Federal Reserve Board, Washington, D.C., 2005.
[10] L. Carver, Mooted VaR substitute cannot be back-tested, says top quant, Risk,
March 08, 2013.
[11] L. Carver, Back-testing expected shortfall: mission impossible?, Risk, October 17,
2014.
[12] J.M. Chen, Measuring market risks under the Basel Accord: VaR, stressed VaR,
and expected shortfall,Aestimatio, The IEB International Journal of Finance, volume
8,(2014) pp. 184-201.
[13] Bank for International Settlements, Fundamental review of the trading book:
A revised market risk framework, Consultative Document, October 2013.
[14] T. Gneiting, Making and Evaluating Point Forecasts, SSRN Preprint, 2010.
[15] A. Graham & J. Pal, Backtesting value-at-risk tail losses on a dynamic portfolio,
Journal of Risk Model Validation, Volume 8, Number 2, 2014.
[16] P. Jorion, Value at Risk : The New Benchmark for Managing Financial Risk, 3rd
Edition, McGraw-Hill, 2007.
[17] J. Kerhof and B. Melenberg, Backtesting for Risk-Based Regulatory Capital,
Journal of Banking and Finance, Vol 28, No 8, (2004) 1845-1865.
[18] W.K. Wong, Backtesting trading risk of commercial banks using expected shortfall,
Journal of Banking & Finance, Volume 32, Issue 7, July 2008, Pages 14041415.
E-mail address: Nick.Costanzino@gmail.com
Risklab Toronto, University of Toronto, 1 Spadina Crescent, Toronto,
ON, M5S 3G3
E-mail address: Michael.Curran@bmo.com
Bank of Montreal, 100 King St W, Toronto, ON, M5X 1A1

11