The CoSamp Algorithm

Sunday, October 12, 2008

Rice University
STAT 631 / ELEC 639: Graphical Models
Instructor: Scribe:
Dr. Volkan Cevher Andrew Waters

1 Motivation
In previous classes, we have explored the topic of compressive sensing. Tra-
ditional techniques for signal acquisition involve acquiring N samples of
a signal sampled at a rate faster than twice the Nyquist rate in order to
guarantee perfect signal reconstruction. Many signals of practical interest,
however, are sparse in some basis, meaning that in some basis they can be
represented with K  N samples. Rather than initially acquire N samples
and then throw away ≈ N − K samples during compression, we would like
to be able to sample the compressed signal directly.

This can be accomplished via compressive sensing techniques to produce

a compressed signal y by taking M ≈ K linear projections of the desired
signal, x, via a projection matrix Φ, as follows:

y = Φx, (1)
where x is an N x1 (sparse) vector, Φ is an M xN matrix, and y is an M x1
vector.

For signal reconstruction to be successful, Φ must satisfy the restricted isom-

etry property or RIP:

(1 − δK )kxk22 ≤ kΦxk22 ≤ (1 + δK )kxk22 , (2)

where k k2 defines the `2 norm. Finding matrices that satisfies this property
is, in general, an NP-complete problem. However, random matrices chosen
with i.i.d random entries (Gaussian, Bernouli, etc) will satisfy the RIP with
high probability subject to M = O(K log(N/K))  N .

Several algorithms exist for performing the signal reconstruction problem.

Some of these include:

1
 • Convex Optimization:

– Basis Pursuit (BP)

– Basis Pursuit De-Noising (BPDN)
– Second-Order Cone Programming

• Iterative Greedy Algorithms

– Matching Pursuit (MP)

– Orthogonal Matching Pursuit (OMP)
– CoSaMP

Here, we will focus our attention on the CoSaMP (Compressive Sampling

Matching Pursuit) algorithm. In section 2 we briefly discuss some notational
conventions that will be used throughout this document. In section 3 we
present the steps of CoSaMP algorithm. Finally, in Section 4 we derive sev-
eral lemmas regarding the convergence properties of the CoSaMP algorithm.
In particular, we will show that the error in the CoSaMP estimate of a given
sparse vector decays exponentially in the number of algorithm iterations.

2 Notation
We define some basic notation used throughout this scribe to eliminate con-
fusion.

Vectors:

-Bolded lower case letters (x, y, etc) are used exclusively for vectors.

-Double bars around a vector with Èan subscript p (ex. kxkp ) indicate the
P p
ell-p norm of the vector x. kxkp = p
i |xi |

Matrices:

-Matrices are denoted with upper-case, bold letter (Φ, etc).

-Φ∗ indicates the Hermitian transpose of Φ

-Φ† indicates the pseudo-inverse of Φ such that Φ† = (Φ∗ Φ)−1 Φ∗

2
Sets:

-Captial letters (ex: T , S) are used to indicate sets.

-|T | indicate the cardinality of the set, or the number of elements that it
contains.

-T C indicates the compliment of set T

-T \ S is used to indicate the set T with all elements in S removed.

-We use the subscribpt notation |T to show that a vector or matrix is being
restricted to only certain elements or columns. For example, x|T indicates
the vector x is restricted only the elements given in T . Φ|T C indicates that
the matrix Φ is restricted to the colums contained in T C .

-We often use the notation xN to indicate the best N -point support set of
the vector x, supp(x). That is, the set of indices that best represent the
vector x by minimizing kx − x|supp(x) k2

3 CoSaMP Algorithm
The following table lists the steps of the CoSaMP algorithm:

3
 1) Initialization:
x−1 = 0 (xJ is the estimate of x) at the J th iteration
r = y (the current residual)
2) Loop until convergence
i) Compute the current error: (Note that for Gaussian Φ, ΦT Φ is ∼ diagonal)
e = Φ∗ r.

ii) Compute the best 2K support set of the error (index set):
Ω = e2K .

iii) Merge the the strongest support sets:

S
T = Ω supp(xJ−1 ).

iv) Perform a Least-Squares Signal Estimation:

b|T = Φ†|T y, b|T c = 0.

v) Prune xJ and compute r for next round:

x J = bk ,
r = y − ΦxJ .

4 Performance Bounds on CoSaMP algorithm

In this section, we prove several important lemmas regarding the CoSaMP
algorithm and use these to prove convergence bounds for the algorithm.
Specifically, we will show that the error in the CoSaMP estimate decays
exponentially until achieving a lower bound dominated primarily by the
noise power present in the measured signal.

4.1 Lemma 1
Suppose that Φ obeys the RIP and has an isometry constant δr . Also, define
T to be a set of r indices. Then:

4
 È
kΦ∗T uk2 ≤ 1 + δr kuk2 , (3)
1
kΦ†T uk2 ≤ √ kuk2 , (4)
1 + δr
È È
1 − δr kuk2 ≤kΦ∗T ΦT uk2 ≤ 1 + δr kuk2 , (5)
1 1
√ kuk2 ≤kΦ∗T ΦT uk2 ≤ √ kuk2 . (6)
1 − δr 1 + δr
Proof: This follows directly from the
√ RIP property
√ which indicates that the
singular values of Φ|T lie between 1 + δr and 1 − δr .

4.2 Lemma 2
T
Suppose that we have index sets S and T that are disjoint (S T = ∅). Now
S
define R = S T with |R| ≤ r.

kΦ∗|S Φ|T k2 ≤ δr . (7)

Proof: This follows from the spectral norm inequality, which states that any
given matrix M has a spectral norm greater than or equal to the spectral
norm of any of its submatrices. For our case, if we define M = Φ|R ∗ Φ|R − I
we can see that:

" #
Φ∗|S Φ|T − I Φ∗|S Φ|T
M= . (8)
Φ∗|T Φ|S Φ∗|T Φ|T − I
Hence, Φ∗|S Φ|T is a submatrix of M. By the spectral norm inequality:

kΦ∗|S Φ|T k ≤ kMk. (9)

(10)

Applying the triangle inequality we also can see that:

kMk ≤ kΦ∗|R Φ|R k + kIk. (11)

We combine the above with the results of Lemma 1 in order to obtain:

5
 kΦ∗|S Φ|T k ≤ kMk, (12)
≤ max{(1 + δr ) − 1, 1 − (1 − δr )}, (13)
= δr . (14)

which concludes the proof.

4.3 Lemma 3
Suppose that Φ obeys the RIP and has isometry constant δr . Let T be a set
S
of indices and x be a vector. Further suppose that r ≥ | T supp(x) | then:

kΦ∗T ΦT x|T c k2 ≤ δr kx|T c k2 . (15)

Proof: This follows directly from the spectral norm inequality with applica-
tion of the results from Lemma 2:

kΦ∗T ΦT x|T C k2 = kΦ∗T (Φ|T C x|T C )k2 , (16)

≤ kΦT ∗ Φ|T C k kx|T C k, (17)
≤ δr kx|T C k. (18)

which concludes the proof.

4.4 Lemma 4
Define:

s = x − xJ
r = y − ΦxJ = Φs + n
e = Φ∗ r and
Ω = e2K

We will now show that:

ks|ΩC k ≤ 0.2223krk2 + 2.34knk2 . (19)

Proof:

6
Defining R = supp(s) we can show that

ke − e|Ω k22 ≤ ke − eR k22 , (20)

X X
⇒ (e(n) − eΩ (n))2 ≤ (e(n) − eR (n))2 , (21)
n n
X X
⇒ (e(n))2 ≤ (e(n))2 , (22)
n∈Ω
/ n∈R
/
X X
⇒ (e(n))2 ≥ (e(n))2 , (23)
n∈Ω n∈R
X X
⇒ (e(n))2 ≥ (e(n))2 , (24)
n∈Ω\R n∈R\Ω
/

⇒ ke|Ω\R k22 ≥ ke|R\Ω k22 . (25)

Expanding the LHS of (25):

ke|Ω\R k22 = kΦ∗|Ω\R (Φs + n) k2 , (26)

≤ kΦ∗|Ω\R Φsk2 + kΦ|Ω\R nk2 , (27)
È
≤ δ4K ksk2 + 1 + δ2K knk2 . (28)

Expanding the RHS (25):

ke|R\Ω k22 = kΦ∗|R\Ω (Φs + n) k2 , (29)

≥ kΦ∗|R\Ω Φsk − kΦ|R\Ω nk2 , (30)
= kΦ∗|R\Ω Φs|R\Ω k − kΦ∗|R\Ω Φs|(R\Ω)C k − kΦ|R\Ω nk2 , (31)
È
≥ (1 − δ2K )ks|R\Ω k2 − δ2K ksk2 − 1 + δ2K knk2 . (32)

Substituting the expansions of both the LHS and RHS into the inital in-
equality we find that:

√
2 1 + δ2K
2.34 = , (33)
1 − δ2K
δ2K + δ4K
.2233 = , (34)
1 − δ4K
δ2K ≤ δ4K ≤ .1. (35)

7
4.5 Lemma 5
Define:

b|T = Φ†|T y
b|T C = 0

then:

kx − bk2 ≤ 1.112kx|T C k2 + 1.06knk2 . (36)

Proof:

kx − bk2 ≤ kx|T C k2 + kx|T − b|T k2 , (37)

€ Š
= kx|T C k2 + kx|T − Φ†|T Φx|T + Φx|T C + n k2 , (38)
€ Š
= Φ†|T Φx|T C + n k2 , (39)
≤ kΦ†|T Φx|T C k2 + kΦ†|T nk2 . (40)
€ Š−1
We now recall that Φ†|T = Φ∗|T Φ|T Φ∗|T and apply the results of Lemma
1 to arrive at:

δ4K knk2
kx − bk2 ≤ kx|T C k2 + √ knk2 , (41)
1 − δ3K 1 − δ3K
= 1.112kx|T C k2 + 1.06knk2 . (42)

4.6 Lemma 6
We wish to show that:

kx − xJ k2 ≤ 2kx − bk2 . (43)

Proof:

We prove this lemma through application of the triangle inequality:

8
 kx − xJ k2 = kx − b + b − xJ k2 , (44)
≤ kx − bk2 + |xJ − bk2 , (45)
≤ 2kx − bk2 . (46)

4.7 Final Result on Convergence

We now continue from the results of Lemma 6, substituting in our previous
results, to show our final result:

kx − xJ k2 ≤ 2kx − bk2 , (47)

≤ 2.224ks|T C k2 + 2.12knk, (48)
≤ 2.224(.2223ksk2 + 2.34knk) + 2.12knk, (49)
≤ .5ksk2 + 7.5knk2 , (50)
= .5kx − xJ−1 k2 + 7.5knk2 . (51)

What this result tells us is that the error in our approximation of x decays
exponenentially each iteration until finally reaching a limit bound by the
noise power present in our signal. This makes the CoSaMP algorithm a
very power tool in sparse signal reconstruction.