Notes on mean embeddings and covariance

operators
Arthur Gretton
February 24, 2015

1 Introduction
This note contains more detailed proofs of certain results in the lecture notes
on mean embeddings and covariance operators. The notes are not as complete
as for lectures 1 and 2, but cover only the trickier concepts. Please let me know
if there are any further parts you’d like clarified, and I’ll add them to the note.

2 Mean embeddings
2.1 Proof that the mean embedding exists via Riesz
For finite dimensional feature spaces, we can define expectations in terms of
inner products.    
x a
φ(x) = k(·, x) = f (·) =
x2 b
Then  >  
a x
f (x) = = hf, φ(x)iF .
b x2
Consider random variable x ∼ P
 >  !  >    >
a x a EP x a
EP f (x) = EP = =: µP .
b x2 b EP (x2 ) b

Does this reasoning translate to infinite dimensions?

Definition 1 (Bounded operator). A linear operator A : F → R is bounded
when
Af ≤ λA kf kF ∀f ∈ F.

We prove via Riesz that the mean embedding exists, and that it takes the
form of the expectation of the canonical map.

1
Theorem 2. [Riesz representation]In a Hilbert space F, all bounded linear
operators A can be written h·, gA iF , for some gA ∈ F,

Af = hf, gA iF

Now we establish the existence of the mean embedding.

p
Lemma 3 (Existence of mean embedding ). If EP k(x, x) < ∞ then µP ∈ F.
Proof. The linear operator TP f := EP f (x) for all f ∈ F is bounded under the
assumption, since
p 
|TP f | = |EP f (x)| ≤ EP |f (x)| = EP |hf, φ(x)iF | ≤ EP k(x, x) kf kF ,
(a)

where in (a) we use Jensen’s inequality. Hence by the Riesz representer theorem
[6, Theorem II.4], there exists a µP ∈ F such that TP f = hf, µP iF .
If we set f = φ(x) = k(x, ·), we obtain µP (x) = hµp , k(x, ·)i = EP k(x, x): in
other words, the mean embedding of the distribution P is the expecation under
P of the canonical feature map.

2.2 Proof that MMD injective for universal kernel

First, it is clear that P = Q implies MMD {P, Q; F } is zero. We now prove the
converse. By the universality of F, for any given  > 0 and f ∈ C(X ) there
exists a g ∈ F such that
kf − gk∞ ≤ .
We will need [2, Lemma 9.3.2 ]:
Lemma 4. Let (X , d) be a metric space, and let P, Q be two Borel probability
measures defined on X , where we define the random variables x ∼ P and y ∼ Q.
Then P = Q if and only if EP (f (x)) = EQ (f (y)) for all f ∈ C(X ), where C(X )
is the space of bounded continuous functions on X .
We now use these two results to formulate a proof. We begin with the
expansion

|EP f (x) − EQ f (y)| ≤ |EP f (x) − EP g(x)|+|EP g(x) − EQ g(y)|+|EQ g(y) − EQ f (y)| .

The first and third terms satisfy

|EP f (x) − EP g(x)| ≤ EP |f (x) − g(x)| ≤ .

Next, write
EP g(x) − EQ g(y) = hg, µP − µQ i = 0,
since MMD {P, Q; F } = 0 implies µP = µQ . Hence

|EP f (x) − EQ f (y)| ≤ 2

for all f ∈ C(X ) and  > 0, which implies P = Q by Lemma 4.

2
3 Covariance operators
One of the most important and widely used tools in RKHS theory is the covari-
ance operator: this is an infinite dimensional analog to the covariance matrix.
This forms the backbone of kernel PCA, the kernel Fisher discriminant, kernel
partial least squares, the kernel canonical correlation, and so on.
In this note, we describe the Hilbert space of Hilbert-Schmidt operators. We
then introduce the covariance operator, demonstrate it is Hilbert-Schmidt, and
express it in terms of kernel functions.

3.1 Hilbert-Schmidt operators

This discussion is based on [9, Section 2.1] and [8, Section A.5.2].
Let F and G be separable Hilbert spaces. Define (ei )i∈I to be an orthonormal
basis for F, and (fj )j∈J to be an orthonormal basis for G. The index sets I, J
are assumed to be either finite our countably infinite.1 Define two compact
linear operators L : G → F and M : G → F. Define the Hilbert-Schmidt norm
of the operators L, M to be
2
X 2
kLkHS = kLfj kF (3.1)
j∈J

hLfj , ei i 2 ,
XX 
= F
i∈I j∈J

where we use Parseval’s indentity on each of the norms in the first sum. The
operator L is Hilbert-Schmidt when this norm is finite.
The Hilbert-Schmidt operators mapping from G to F form a Hilbert space,
written HS(G, F), with inner product
X
hL, M iHS = hLfj , M fj iF , (3.2)
j∈J

which is independent of the orthonormal basis chosen. It is clear the norm (3.1)
is recovered from this inner product. Another form for this inner product is
XX
hL, M iHS = hLfj , ei iF hM fj , ei iF . (3.3)
i∈I j∈J

Proof. Since any element of F can be expanded in terms of its orthonormal

basis, we have that this holds in the specific case of the mapping of fj by L or
M, X (j) X (j)
Lfj = αi ei M fj = βi0 ei0 . (3.4)
i∈I i0 ∈I

1 Recall that a Hilbert space has a countable orthonormal basis if and only if it is separable:

that is, it has a countable dense subset [6, p. 47].

3
Substituting these into (3.2), we obtain
* +
X X (j) X (j)
hL, M iHS = αi ei , βi0 e i0
j∈J i∈I i0 ∈I F
(j) (j)
XX
= αi βi .
i∈I j∈J

We obtain the identical result when we substitute (3.4) into (3.3).

3.2 Rank-one operators, tensor product space

Given b ∈ G and a ∈ F, we define the tensor product a⊗b as a rank-one operator
from G to F,
(b ⊗ a)f 7→ hf, aiF b. (3.5)
First, is this operator Hilbert-Schmidt? We compute its norm according to
(3.1),

2
X 2
ka ⊗ bkHS = k(a ⊗ b)fj kF
j∈J

a hb, fj i 2
X
= G F
j∈J

hb, fj i 2
X
= kak2F

G
j∈J

= kak2F kbk2G , (3.6)

where we use Parseval’s identity. Thus, the operator is Hilbert-Schmidt.

Given a second Hilbert-Schmidt operator L ∈ HS(G, F), we have the result:

hL, a ⊗ biHS = ha, LbiF (3.7)

A particular instance of this result is

hu ⊗ v, a ⊗ biHS = hu, aiF hb, viG . (3.8)

Proof. ThePkey result we use is the expansion of b in terms of the orthonormal

basis, b = j∈J hb, fj iG fj . Then
*  +
X
ha, Lbi = a, L  hb, fj iG fj 
j
F
X
= hb, fj iG ha, Lfj iF
j

4
and
X
ha ⊗ b, LiHS = hLfj , (a ⊗ b)fj iF
j
X
= hb, fj iG hLfj , aiF .
j

To show (3.8), we simply substitute u ⊗ v for L above, and then apply the
definition (3.5),

hu ⊗ v, a ⊗ biHS ha, (u ⊗ v) biF

= hu, aiF hb, viG

3.3 Cross-covariance operator

In this section, we define the cross-covariance operator, in the case where F and
G are reproducing kernel Hilbert spaces with respective kernels k and l, and
feature maps φ and ψ. This is a generalization of the cross-covariance matrix
to infinite dimensional feature spaces. The results we want are feature space
analogues to:
eXY = E xy> f >C
eXY g = Exy f > x g > y ,




C

where we use the notation CeXY to denote a covariance operator without cen-
tering. The corresponding centered covariance is
eXY − µX µ>
CXY := C Y,

where µX := E(x) and µY := E(y)We now describe how we can get these results
in feature space.
The cross product φ(x)⊗ψ(y) is a random variable in HS(G, F): use the result
in [9, p. 265] that for all A ∈ HS(G, F), the linear form hφ(x) ⊗ ψ(y), AiHS is
measurable. For the expectation of this random variable to exist (and to be
an element of HS(G, F)), we require the expected norm of φ(x) ⊗ ψ(y) to be
bounded: in other words, Ex,y (kφ(x) ⊗ ψ(y)kHS ) < ∞. Given the expectation
exists, and writing it C
eXY , then this expectation is the unique element satisfying
D E
CeXY , A = Ex,y hφ(x) ⊗ ψ(y), AiHS (3.9)
HS

Proof. The operator

Txy : HS(G, F) → R
A 7→ Ex,y hφ(x) ⊗ ψ(y), AiHS

is bounded when Ex,y (kφ(x) ⊗ ψ(y)kHS ) < ∞, since by applying first Jensen’s
inequality, then Cauchy-Schwarz,

5
 |Ex,y hφ(x) ⊗ ψ(y), AiHS | ≤ Ex,y |hφ(x) ⊗ ψ(y), AiHS |
≤ kAkHS Ex,y (kφ(x) ⊗ ψ(y)kHS ) .
Thus by the Riesz representer theorem (Theorem (2)), the covariance operator
(3.9) exists. We can make a further simplification to the condition: substituting
(3.6), we get the requirement
Ex,y (kφ(x) ⊗ ψ(y)kHS ) = Ex,y (kφ(x)kF kφ(y)kG )
p 
= Ex,y k(x, x)l(y, y) < ∞.

We could also use the weaker condition Ex,y (k(x, x)l(y, y)), which is implied from
the above by Jensen’s inequality.
We now use the particular element f ⊗ g. Combining (3.7) and (3.9), we
have the result
D E D E
f, C
eXY g = eXY , f ⊗ g
C
F HS
= Ex,y hφ(x) ⊗ ψ(y), f ⊗ giHS
 
= Exy hf, φ(x)iF hg, ψ(y)iG
= Exy [f (x)g(y)] = cov(f, g).
What does this operator look like? To see this, we apply it to k(x, ·)l(y, ·) (just
as we plotted the mean embedding by evaluating it on k(x, ·)).
We are given an i.i.d. sample from P = Px Py , written z := ((x1 , y1 ) . . . (xn , yn )).
Write the empirical
n
bXY := 1
X
C φ(xi ) ⊗ ψ(yi ) − µ̂x ⊗ µ̂y ,
n i=1
1
Pn
where we have now included the centering terms µ̂x := n i=1 φ(xi ). With
some algebra, this can be written

bXY = 1 XHY > ,

C
n
where H = In − n−1 1n , and 1n is an n × n matrix of ones, and
   
X = φ(x1 ) . . . φ(xn ) Y = ψ(y1 ) . . . ψ(yn ) .
Define the kernel matrices
Kij = X > X


ij
= k(xi , xj ) Lij = l(yi , yj ),

and the kernel matrices between centred variables,

K
e = HKH L
e = HLH

(exercise: prove that the above are kernel matrices for the variables centred in
feature space).

6
4 Using the covariance operator to detect depen-
dence
There are two measures of dependence we consider: the constrained covariance
(COCO), which is the largest singular value of the covariance operator, and the
Hilbert-Schmidt Independence Criterion, which is its Hilbert-Schmidt norm.

4.1 Empirical COCO and proof

We now derive the functions satisying
D E
maximize g, C
bXY f
G
subject to kf kF = 1 (4.1)
kgkG = 1 (4.2)

We assume that
n
X n
X
f= αi [φ(xi ) − µ̂x ] = XHα g= βi [ψ(yi ) − µ̂y ] = Y Hβ,
i=1 j=1

where
n n
1X 1X
µ̂x = φ(xi ) µ̂y = φ(yi ).
n i=1 n i=1
The associated Lagrangian is
λ
γ
L(f, g, λ, γ) = f > C kf k2F − 1 − kgk2G − 1 ,


bXY g −
2 2
where we divide the Lagrange multipliers by 2 to simplify the discussion later.
We now write this in terms of α and β:
1 >
f >C α HX > XHY > Y Hβ


bXY g =
n
1 > ee
= α K Lβ,
n
where we note that H = HH. Similarly

kf k2F = α> HXX > Hα = α> Kα.

e

Substituting these into the Lagrangian, we get a new optimization in terms of

α and β,
1 > ee λ >e  γ 
L(α, β, λ, γ) = α K Lβ − α Kα − 1 − β > Lβ
e −1 . (4.3)
n 2 2

7
We must maximize this wrt the primal variables α, β. Differentiating wrt α and
β and setting the resulting expressions to zero,2 we obtain
1 ee
K Lβ − λKα
e = 0 (4.4)
n
1ee
LKα − γ Lβ
e = 0 (4.5)
n
Multiply the first equation by α> , and the second by β > ,
1 > ee
α K Lβ = λα> Kα
e
n
1 >e e
β LKα = γβ > Lβ
e
n
Subtracting the first expression from the second, we get

λα> Kα
e = γβ > Lβ.
e

Recall the constraints α> Kα

e = 1 and β > Lβe = 1. Thus for λ 6= 0 and γ 6= 0,
we conclude that λ = γ. Making this replacement in (4.4) and (4.5), we must
maximize the following expression wrt α, β:
" # " #
1 ee
 
0 n K L α K
e 0 α
1ee =γ . (4.6)
n LK 0 β 0 L
e β

This is a generalized eigenvalue problem, and can be solved straightforwardly in

Matlab. The maximum eigenvalue is indeed COCO: at the solution, α> Kα e =1
>e
and β Lβ = 1, hence the two norm terms in the Lagrangian (4.3) vanish.3
2 We use [5, eqs. (61) and (73)]
∂a> U a ∂v > a ∂a> v
= (U + U > )a, = = v.
∂a ∂a ∂a

3 Fora more roundabout way of reaching the same conclusion: pre-multiply (4.6) by [α> β > ]
to get the system of equations
" # " #
1 > ee 1 > e  
n
α K Lβ n
α Kα 1
1 >e e = γ 1 >e = γ ,
β LKα β Lβ 1
n n

where in the final line we substitute the constraints from (4.1).

8
4.2 The Hilbert-Schmidt Independence Criterion
4.2.1 Population expression
What is the Hilbert-Schmidt norm of the covariance operator?4 Consider the
centered, squared norm of the RKHS covariance operator,

HSIC 2 (F, G, Pxy ) = kCeXY − µX ⊗ µY k2HS

D E D E
= C eXY , C
eXY + hµX ⊗ µY , µX ⊗ µY iHS − 2 CeXY , µX ⊗ µY ,
HS HS

where CeXY is the uncentered covariance operator defined in (3.9). There are
three terms in the expansion.
To obtain the first term, we apply (3.9) twice, denoting by (x0 , y0 ) an inde-
pendent copy of the pair of variables (x, y),
D E
kCeXY k2HS = C
eXY , C
eXY
HS
D E
= Ex,y φ(x) ⊗ ψ(y), C eXY
HS
= Ex,y Ex0 ,y0 hφ(x) ⊗ ψ(y), φ(x0 ) ⊗ ψ(y0 )iHS
= Ex,y Ex0 ,y0 hφ(x), [φ(x0 ) ⊗ ψ(y0 )]ψ(y)iF
Ex,y Ex0 ,y0 hφ(x), φ(x0 )iF hψ(y0 ), ψ(y)iG
 
=
= Ex,y Ex0 ,y0 k(x, x0 )l(y, y0 )
=: A

Similar reasoning can be used to show

hµX ⊗ µY , µX ⊗ µY iHS = hµX , µX iF hµY , µY iG

= Exx0 k(x, x0 )Eyy0 l(y, y0 )
=: D,

and for the cross-terms,

D E
CeXY , µX ⊗ µY = Ex,y hφ(x) ⊗ ψ(y), µX ⊗ µY iHS
HS


= Ex,y hφ(x), µX iF hφ(y), µY iG
= Ex,y (Ex0 k(x, x0 )Ey0 l(y, y0 ))
=: B.
4 Other norms of the operator may also be used in determining dependence, e.g. the

spectral norm from the previous section. Another statistic on the kernel spectrum is the
Kernel Mutual Information, which is an upper bound on the true mutual information near
independence, but is otherwise difficult to interpret [4]. One can also define independence
statistics on the correlation operator [1], which may be better behaved for small sample sizes,
although the asymptotic behavior is harder to analyze.

9
4.2.2 Biased estimate
A biased estimate of HSIC was given in [3]. We observe a sample Z :=
{(x1 , y1 ), . . . (xn , yn )} drawn independently and identically from Pxy , we wish
to obtain empirical expressions for HSIC,

HSIC 2 (F, G, Z) := A
b − 2B
b + C.
b

A direct approach would be to replace the population uncentred covariance

operator C
eXY with an empirical counterpart,
n
1X
ČXY = φ(xi ) ⊗ ψ(yi ),
n i=1

and the population mean embeddings with their respective empirical estimates,
n n
1X 1X
µ̂x = φ(xi ), µ̂y = ψ(yi ),
n i=1 n i=1

however the resulting estimates are biased (we will show the amount of bias in
the next section). The first term is
* n n
+
2 1 X 1 X
bb = ČXY =
A φ(xi ) ⊗ ψ(yi ), φ(xi ) ⊗ ψ(yi )
n i=1 n i=1
HS
n n
1 XX 1
= 2 kij lij = 2 tr(KL),
n i=1 j=1 n

we use the shorthand kij = k(xi , xj ), and the subscript b to denote a biased
estimate. The expression is not computationally efficient, and is written this
way for later use - in practice, we would never take the matrix product if the
intent was then to compute the trace. Next,

* n n
! n
!+

bb = ČXY , µ̂X ⊗ µ̂Y

 1X 1X 1X
B = φ(xi ) ⊗ ψ(yi ), φ(xi ) ⊗ ψ(yi )
n i=1 n i=1 n i=1
n
* n
!+ * n
!+HS
1 X 1 X 1 X
= φ(xi ), φ(xi ) ψ(yi ), ψ(yi )
n i=1 n i=1 n i=1
F G
n n n
1 XXX
= 3 kij liq
n i=1 j=1 q=1
1 > 1
= 1 KL1n = 3 1> LK1n
n3 n n n

10
(we will use both forms to get our final biased estimate of HSIC), and
* n
! n
! n
! n
!+
1X 1X 1X 1X
Db = hµ̂X ⊗ µ̂Y , µ̂X ⊗ µ̂Y i =
b φ(xi ) ⊗ ψ(yi ) , φ(xi ) ⊗ ψ(yi )
n i=1 n i=1 n i=1 n i=1
HS
  
n X n n X n
1 X X
= 4 k(xi , xj )  l(yi , yj )
n i=1 j=1 i=1 j=1

1
1>

>

= 4 n K1n 1n L1n
n
We now combine these terms, to obtain the biased estimate
 
1 2 1
HSICb2 (F, G, Z) = 2 tr(KL) − 1> >

>

KL1 n + 1 K1 n 1 L1 n
n n n n2 n n
 
1 1
1 1
= 2 tr(KL) − tr 1n 1> > > >



n KL − tr K1 1
n n L + tr 1 1
n n K1 1
n n L
n n n n2
    
1 1 1
= 2 tr I − 1n 1> n K I − 1n 1> n L
n n n
1
= 2 tr(KHLH)
n
where we define
1
H := I − 1n 1>
n
n
as a centering matrix (when pre-multiplied by a matrix it centers the rows; when
post-multiplied, it centers the columns).

4.2.3 Unbiased estimate

eXY k2 is
An unbiased estimate of A := kC HS

n n
1 XX 1 X
A
b := kij lij = kij lij ,
n(n − 1) i=1 (n)2
j6=i (i,j)∈im
2

where inp is the set of all p-tuples drawn from {1, . . . , n}, and

n! 1
(n)p = = .
(n − p)! n(n − 1) . . . (n − p + 1)

Note that E(A)b = Ex,y Ex0 ,y0 k(x, x0 )l(y, y0 ), which is not true of the biased ex-
pression (which does not properly treat the independent copies x0 of x and y0 of

11
y). The difference between the biased and unbiased estimates is
n
b= 1 1
X X
bb − A
A kij lij − kij lij
n2 i,j=1 n(n − 1)
(i,j)∈im
2
 
n  
1 X 1 1 X
= 2 kii lii − −  kij lij 
n i=1 n2 n(n − 1) m
(i,j)∈i2
 
n
1 1 X 1 X
= kii lii − kij lij  ,
n n i=1 n(n − 1) m (i,j)∈i2

thus the expectation of this difference (i.e., the bias) is O(n−1 ).

The unbiased estimates of the remaining two terms are
1 X
B
b := kij liq
(n)3
(i,j,q)∈in
3

and
1 X
D
b := kij lqr .
(n)4
(i,j,q,r)∈in
4

While these expressions are unbiased, they are at first sight much more expensive
to compute than the respective biased estimates, with B b costing O(n3 ) and D
b
4
costing O(n ). It is possible, however, to obtain these unbiased estimates in
O(n2 ), i.e., the same cost as the biased estimates, as shown by [7, Theorem 2].
First, we note that diagonal entries of the kernel matrices K and L never appear
in the sums, hence we immediately replace these matrices with K e and L
e having
the diagonal terms set to zero. The term A can be written concisely in matrix
b
form as
b= 1 1
   
A e L
K e = trace K eL
e ,
(n)2 ++ (n)2
where  is the entrywise matrix product and (A)++ is the sum of all the entries
in A. Looking next at the term B, b and defining as 1n the n × 1 vector of ones,
we have
 
n n n X n
1 X 1 X X X
Bb= kij liq =  kiq lqj − kiq liq 
(n)3 n
(n)3 i,j=1 i=1
(i,j,q)∈i3 q6=(i,j) q6=i
" Pn Pn #
1 > j=2 k1q lq1 . . . q6=(i,j) kiq lqj ...
= 1 .. .. 1n
(n)3 n . .
1  e e
− K L
(n)3 ++
1 > 1  e e
= 1n K̃ L̃1n − K L .
(n)3 (n)3 ++

12
The first expression in the final line can be computed in time O(n2 ), as long as
the matrix-vector products are taken first. Finally, looking at the fourth term,5
"
1 X 1 X X
D=
b kij lqr = kij lqr
(n)4 (n)4
(i,j,q,r)∈in4 (i,j)∈in n
2 (q,r)∈i2
X X X
− kij lir − kij ljr − kij lij
(i,j,r)∈in
3 (i,j,r)∈in
3 (i,j)∈in
2
| {z } | {z } | {z }
q=i q=j (q=i,r=j)≡(q=j,r=i)
#
X X X
− kij liq − kij ljq − kij lij
(i,j,q)∈in
3 (i,j,q)∈in
3 (i,j)∈in
2
| {z } | {z } | {z }
r=i r=j (r=i,q=j)≡(r=j,q=i)
   
n n n n
1 X X  X X   
= kij lij − 41>
n K̃ L̃1n + 2 K  L
e e 
(n)4 i=1 j6=i i=1 j6=i
++
  
1    
= 1>
n
e n 1> L1
K1 n
e n − 41> K̃ L̃1n + 2 K
n
e Le ,
(n)4 ++

which can also be computed in O(n2 ). We now establish the net contribution
of each term:

e L
 1 2 2
K e : + +
++ (n)2 (n)3 (n)4
(n − 2)(n − 3) + (2n − 6) + 2
=
(n)4
(n − 2)(n − 1)
=
(n)4

and
−2 4
1>
n K̃ L̃1n : −
(n)3 (n)4
−2(n − 3) − 4 −2(n − 1)
= = .
(n)4 (n)4

Thus, we have our empirical unbiased HSIC expression,

 
2 1  2 > 1 
> e

>e
HSIC (F, G, Z) := K L
e e − 1 K̃ L̃1n + 1 K1n 1n L1n
n(n − 3) ++ (n − 2) n (n − 1)(n − 2) n
5 The equivalences ≡ in the first line below indicate that both index matching constraints

amount to the same thing, hence these terms appear only once.

13
5 HSIC for feature selection
As we saw in the previous section, a biased estimate for the centred HSIC can
be written
1
HSIC := 2 trace(KHLH).
n
Consider the case where we wish to find a subset of features that maximizes
HSIC with respect to some set of labels. Assume we have a sample {xi , yi }ni=1 ,
where xi ∈ Rd , and
 binary−1class
labels. We choose a particular form for the
class labels: yi ∈ n−1
+ , −n− , where n+ is the number of positive labels and
n− is the number of negative labels.
We denote by xi [`] the `th coordinate of xi , and write
 >
x[`] := x1 [`] . . . xn [`]

the column vector of the `th coordinate of all samples. If we use a linear kernel
on the xi , then
X d
Ki,j = x> x
i j = xi [`]xj [`].
`=1

It follows we can write the kernel as the sum of kernels on individual dimensions,
d
X
K= K` ,
`=1

where K` := x[`]x[`]> . In this case, HSIC is the sum of HSIC values for each
such kernel,
d
1 X
HSIC := 2 trace(K` HLH).
n
`=1

What happens when we choose a linear kernel on the labels? Assuming the
classes are grouped together,

n−2
 
> + I −n+ n− I
L = yy = ,
−n+ n− I n−2
− I

where y is the vector of all class labels. Note further than

n
X
yi = 0,
i=1

14
and hence HLH = L. Finally, using trace(AB) = trace(BA),
d
1 X
HSIC = 2 trace(K` L)
n
`=1
d
1 X
= 2 trace(x[`]x[`]> yy > )
n
`=1
 2
d n+ n
1 X
 1
X 1 X
= 2 xi [l] − xi [l]
n n+ i=1 n− i=n+ +1
`=1

6 Acknowledgments
Thanks to Aaditya Ramdas, Wittawat Jitkrittum, and Dino Sejdinovic for cor-
rections and improvements to these notes.

References
[1] F. R. Bach and M. I. Jordan. Kernel independent component analysis.
Journal of Machine Learning Research, 3:1–48, 2002.
[2] R. M. Dudley. Real analysis and probability. Cambridge University Press,
Cambridge, UK, 2002.
[3] A. Gretton, O. Bousquet, A. J. Smola, and B. Schölkopf. Measuring sta-
tistical dependence with Hilbert-Schmidt norms. In Algorithmic Learning
Theory: 16th International Conference, pages 63–78, 2005.
[4] A. Gretton, R. Herbrich, A. J. Smola, O. Bousquet, and B. Schölkopf. Ker-
nel methods for measuring independence. Journal of Machine Learning Re-
search, 6:2075–2129, 2005.
[5] K. B. Petersen and M. S. Pedersen. The matrix cookbook, 2008. Version
20081110.
[6] M. Reed and B. Simon. Methods of modern mathematical physics. Vol. 1:
Functional Analysis. Academic Press, San Diego, 1980.
[7] L. Song, A. Smola, A. Gretton, J. Bedo, and K. Borgwardt. Feature selection
via dependence maximization. JMLR, 13:1393–1434, 2012.
[8] Ingo Steinwart and Andreas Christmann. Support Vector Machines. Infor-
mation Science and Statistics. Springer, 2008.
[9] L. Zwald, O. Bousquet, and G. Blanchard. Statistical properties of kernel
principal component analysis. In Proc. Annual Conf. Computational Learn-
ing Theory, 2004.

15