IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 21, NO.

1, JANUARY 2012

393

Edge Strength Filter Based Color Filter Array Interpolation

Ibrahim Pekkucuksen, Student Member, IEEE, and
Yucel Altunbasak, Senior Member, IEEE

AbstractFor economic reasons, most digital cameras use color lter arrays instead of beam splitters to capture image data. As a result of this, only
one of the required three color samples becomes available at each pixel location and the other two need to be interpolated. This process is called Color
Filter Array (CFA) interpolation or demosaicing. Many demosaicing algorithms have been introduced over the years to improve subjective and objective interpolation quality. We propose an orientation-free edge strength
lter and apply it to the demosaicing problem. Edge strength lter output
is utilized both to improve the initial green channel interpolation and to
apply the constant color difference rule adaptively. This simple edge directed method yields visually pleasing results with high CPSNR.
Index TermsColor lter array (CFA) interpolation, demosaicing, edgedirected interpolation, orientation-free edge lter.

I. INTRODUCTION
Color images require multiple data samples for each pixel as opposed
to grayscale images for which a pixel is represented by only one data
sample. For the RGB image format, these data samples represent red,
green and blue channels . A typical digital camera captures only one
of these channels at each pixel location and the other two need to be
estimated to generate the complete color information. This process is
called color lter array (CFA) interpolation or demosaicing. Although
many different CFA patterns have been proposed, the most prevalent
one is the Bayer pattern shown in Fig. 1 [1].
As an important step in image processing pipeline of digital cameras,
demosaicing has been an area of interest in both academia and industry.
The simplest approach to the demosaicing problem is to treat color
channels seperately and ll in missing pixels in each channel using
a spatially invariant interpolation method such as bilinear or bicubic
interpolation. While such an approach works ne in homogenous areas,
it leads to color artifacts and lower resolution in regions with texture
and edge structures.
Obtaining better demosaicing performance is possible by exploiting
the correlation between the color channels. Spectral correlation can be
modeled by either constant color ratio rule [2], [3] or constant color difference rule [4], [5]. The basic assumption is that color ratio/difference
is constant over a local distance inside a given object. This assumption
is likely to break apart across boundaries, hence many demosaicing algorithms try to utilize it adaptively in one way or another.
Since the Bayer CFA pattern has twice as many green channel samples as red and blue ones, green channel suffers less from aliasing and is
the natural choice as the starting point of the CFA interpolation process.
In [6], Glotzbach et al. proposed improving red and blue channel interpolation by adding high frequency components extracted from green
channel to red and blue channels. In another frequency-domain approach, Gunturk et al. [7] used an alternating projections scheme based
Manuscript received August 03, 2010; revised December 30, 2010; accepted
April 28, 2011. Date of publication May 19, 2011; date of current version December 16, 2011. The associate editor coordinating the review of this manuscript
and approving it for publication was Prof. Kenneth KM Lam.
The authors are with the Department of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, GA 30332 USA (e-mail:
ibrahimp@gatech.edu; yucel@ece.gatech.edu).
Color versions of one or more of the gures in this paper are available online
at http://ieeexplore.ieee.org.
Digital Object Identier 10.1109/TIP.2011.2155073

Fig. 1. Bayer CFA pattern.

on strong inter-channel correlation in high frequency subbands. Although the main objective is to rene red and blue channels iteratively,
the same approach can also improve green channel interpolation beforehand which in turn yields better red and blue channel results. A
more recent method [8] makes several observations about color channel
frequencies and suggests that ltering the CFA image as a whole instead of individual color channels should preserve high frequency information better. To estimate luminance, the method proposes a xed
5-by-5 lter at green pixel locations and an adaptive lter for red and
blue pixel locations. The estimated full resolution luminance is then
used to complete missing the chrominance information.
Edge-directed green channel interpolation has been proposed early
on with various direction decision rules [4], [5], [9], [10]. The method
outlined in [4] is particularly noteworthy because it proposed using
derivatives of chrominance samples in initial green channel interpolation. Several subsequent demosaicing algorithms made use of this idea.
Authors of [11] proposed using variance of color differences as a decision rule while Zhang et al. [12] proposed making a soft decision to
improve the interpolation performance of the original method [4]. In
this method [12], color differences along horizontal and vertical directions are treated as noisy observations of the target pixel color difference and they are combined optimally using the linear minimum mean
square error estimation (LMMSE) framework. Paliy et al. [13] further
improved directional ltering proposed in [12] by introducing scale
adaptive ltering based on linear polynomial approximation (LPA).
Several methods proposed performing interpolation in both horizontal and vertical directions and making a posteriori decision based
on some criteria. Hirakawa et al. [15] compared local homogeneity of
horizontal and vertical interpolation results and Menon et al. [16] used
color gradients over a local window to make the direction decision.
The rest of the paper is organized as follows. Section II describes
the proposed CFA interpolation algorithm. Section III presents experimental results, and Section IV reports the conclusions.
II. PROPOSED ALGORITHM
The basis of the proposed algorithm is the observation that the
constant color difference assumption tends to fail across edges. If one
can effectively utilize edge information to avoid averaging non-correlated color differences, demosaicing performance could increase
dramatically.
The question at this point is, how the edge information can be expressed meaningfully at the pixel level so that it is useful enough to
improve demosaicing performance. Edge detection lters such as Sobel
and Canny can tell whether an edge structure is present at a given pixel.
However, they do not provide any information about the sharpness of
luminance transition at that particular pixel.
We propose an edge strength lter that provides local, orientationfree luminance transition information. The lter has a 3 by 3 support
size. Given a grayscale input image, it could be formulated as

SP 6 =

P1 0 P11 j

1057-7149/$26.00 2011 IEEE

P 3 0 P9 j

P2 0 P10 j + jP5 0 P7 j

(1)

394

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 21, NO. 1, JANUARY 2012

where SP 6 stands for the edge strength at pixel location P6 .

By applying the lter to all available pixels, we get the edge strength
map of the input image. Note that, although the lter result for a single
pixel does not provide any edge direction information, the relationship
between neighboring pixel results yields the edge orientation in that
neighborhood.
The proposed lter is very useful for nding edges in a grayscale
image. However, a mosaicked image only has one of the three color
channels available for every pixel location and it certainly does not have
complete luminance information at any pixel. That is why, the edge
strength lter can only be applied to a mosaicked image by making an
approximation. Instead of trying to estimate luminance information and
taking estimated luminance differences of neighboring pixel pairs, we
take the difference in terms of the available color channel for each pixel
pair. For instance, for the red center pixel case the diagonal differences
will come from the blue channel and the rest from the green channel

SR

B5 0 B15 j

B7 0 B13 j

+ G6
j

G14 j + jG9 0 G11 j:

(2)

The edge strength for green and blue pixels will be calculated in the
same way. The edge strength map obtained from the mosaicked input
image will help us both in initial green channel interpolation stage and
in subsequent green channel update.
A. Green Channel Interpolation
We propose making a hard decision based on the edge strength lter
described above. For this purpose, every green pixel to be interpolated
(red or blue pixel in the mosaicked image) is marked either horizontal
or vertical by comparing the edge strength differences along each direction on a local window. For a window size of 5 by 5, horizontal and
vertical difference costs can be formulated as follows:

Hi;j

Vi;j

(Si+m;j+n

m=02 n=02
1

(Si+m;j+n

m=02 n=02

Si+m;j +n+1 )
Si+m+1;j +n )

(3)

where Si;j is the edge strength lter output at pixel location (i; j ), and
and Vi;j represent the total horizontal and vertical costs, respectively.
The target pixel will be labeled horizontal if horizontal cost is less
than vertical and vice versa. The rationale behind this decision scheme
is that if there happens to be a horizontal edge in a given neighborhood,
then the edge strength differences between vertical neighbors will vary
more than those of horizontal neighbors. After all the pixels are labeled,
the robustness of the direction decision can be improved by relabeling
them based on the directions of their neighbors. For instance, considering the closest 8 neighbors of a target pixel and the pixel itself, it will
be labeled horizontal only if more than 4 of those 9 pixels are initially
labeled horizontal.
Based on the nal direction label, green channel is interpolated as
follows:

Hi;j

~
G 0B
2
~
+ G 0B
4
~
G 0B
Bi;j +
2
~
G
0B

Bi;j +

~
Gi;j

~
0B

+G
+

;
G

if Horizontal

~
0B
4

if Vertical

where directional estimations are calculated by

~ i;j
GH

= Gi;j01 + Gi;j+1 + 2
2

Bi;j 0 Bi;j 02 0 Bi;j +2

(4)

~ i;j
GV
~H
Bi;j
~V
Bi;j

= Gi01;j + Gi+1;j + 2
2
Bi;j 01 + Bi;j +1 2
=
+
2
= Bi01;j + Bi+1;j + 2
2

Bi;j 0 Bi02;j 0 Bi+2;j

Gi;j 0 Gi;j 02 0 Gi;j +2

Gi;j 0

4
4
Gi02;j
4

Gi+2;j
:

(5)

Green channel estimation for red pixel locations is performed simply

by replacing B s with Rs in the equations above.
B. Green Channel Update
The second step of the proposed algorithm is updating the green
channel. We make use of the constant color difference assumption combined with edge strength lter to improve the initial green channel interpolation while avoiding averaging across edge structures. For every
green pixel to be updated, the closest four neighbors with available
color difference estimates are considered. We expect the edge strength
difference between two pixels to be large across edges. That is why the
weight for each neighbor is inversely correlated with the total absolute
edge strength difference in its direction. In other words, a neighbor will
contribute less to the update result if there happens to be a strong edge
between the target pixel and itself. Assuming we are updating the green
channel value at a blue pixel:

D1 = jSi;j 0 Si01;j j + jSi01;j 0 Si02;j j

+ jSi02;j 0 Si03;j j + C1
D2 = jSi;j 0 Si;j 01 j + jSi;j 01 0 Si;j 02 j
+ jSi;j02 0 Si;j03 j + C1
D3 = jSi;j 0 Si;j +1 j + jSi;j +1 0 Si;j +2 j
+ jSi;j+2 0 Si;j+3 j + C1
D4 = jSi;j 0 Si+1;j j + jSi+1;j 0 Si+2;j j
+ jSi+2;j 0 Si+3;j j + C1
M1 = D2 3 D3 3 D4 M2 = D1 3 D3 3 D4
M3 = D1 3 D2 3 D4 M4 = D1 3 D2 3 D3
^i;j = Bi;j + W 3 (Gi;j 0 Bi;j ) + (1 0 W )
~
G
M1 ~
(G 0 Bi02;j )
3
MTotal i02;j
~
+ MM2 (Gi;j02 0 Bi;j02 )
Total
~
+ MM3 (Gi;j+2 0 Bi;j+2 )
Total
~
+ MM4 (Gi+2;j 0 Bi+2;j )
Total
MTotal = M1 + M2 + M3 + M4 :

(6)

Again, green channel values at red pixel locations are updated in

^
the same way by replacing B s with Rs in the equations above. Gi;j
~
stands for updated green channel result while Gi;j is the initial green
channel interpolation. C1 is a nonzero constant to avoid zero denominator. W1 is the weight for the initial color difference estimation and
W2 is the neighbors contribution to the green channel update. Updating green channel reduces color artifacts and improves PSNR. However, zipper artifacts become more prominent as the number of updates
increase. Experiments on test images suggest that one or two green
channel updates are adequate.
The performance of green channel update can be improved further
by making W1 adaptive for each pixel by checking the total absolute
difference between the closest known green pixels. The idea is that
green channel update should be more aggresive if there happens to be
a lot of difference between known green pixels in that neighborhood

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 21, NO. 1, JANUARY 2012

395

because initial interpolation is more likely to fail in such areas. The

update equations with adaptive weights are shown in (7) at the bottom
of the page.

TABLE I
COMPARISON OF CPSNR VALUES FOR DIFFERENT DEMOSAICING METHODS.

C. Red and Blue Channel Interpolation

Once the green channel interpolation is nalized, we ll in red and
blue channels using constant color difference assumption. For red
channel interpolation at blue pixels and blue channel interpolation
at red pixels, diagonal neighbors are used adaptively based on green
channel gradients in both directions

^
^
^
^
M1 = G 02 02 G + G 01 01 G +1 +1
^ G +2 +2
^
+G
^
^
^
^
M2 = G 02 +2 G + G 01 +1 G +1 01
^ G +2 02 :
^
(8)
+G
If coordinate (i;j ) is a red pixel location, blue channel estimation is
j

i
j

;j

i;j 0

i
j

i;j 0

;j

;j

i;j j

;j

;j

;j

;j

i;j j

;j

TABLE II
COMPARISON OF S-CIELAB VALUES FOR DIFFERENT DEMOSAICING
METHODS.

calculated by

^
B =
^
G
^
G M2 (G 01 01 B 0(1M01++M^2)+1 +1 B +1 +1)
2 1
^
^
M1 (G 01 +1 B 01 +1 + G +1 01 B +1 01) :
2 (M1 + M2 )
i;j

i;j 0

;j

;j

;j

;j

;j

;j

;j

;j

(9)

The equations are similar for red channel estimation at a blue pixel
location.
For red and blue channel estimation at green pixels, we employ bilinear interpolation over color differences since considered adaptive approaches do not provide any performance gain. Here, only the closest
two neighbors for which the original pixel value available are used

^
(G2 01 2

^
B2 01 2 )+(G2 +1 2 B2 +1 2 )

^
B2 2 =G2 2
2
^
B2 +1 2 +1 =G2 +1 2 +1
^
^
(G2 +1 2 B2 +1 2 )+(G2 +1 2 +2 B2 +1 2 +2 ) :
i; j 0

i; j

; j

; j 0

; j

; j0

; j

; j
i

; j0

; j

; j

; j

By the end of this step, we lled in all the missing color channel
values in the input image. We utilized a simple edge strength lter
both to determine the initial green channel interpolation direction and
to avoid applying constant color difference rule across edge structures.
III. EXPERIMENTAL RESULTS
The proposed algorithm is tested on the Kodak image set that was
used in a recent survey paper [14]. The test set consists of 12 images
with 512-by-768 pixel resolution. The images are rst downsampled in
Bayer CFA pattern and then interpolated back to three channels using

proposed algorithm. The interpolated images are compared to the original images and results are reported in terms of CPSNR error measure.
Pixels within 10 pixel distance from the border are excluded from the
calculations.
Results of the proposed algorithm are compared with Alternating
Projections (AP) [7],Variance of Color-Difference (VCD) [11], Directional Linear Minimum Mean Square-Error Estimation (DLMMSE)
[12], Local Polynomial Approximation (LPA) [13], Regularization
Approaches to Demosaicking (RAD) [17], Gradient Based Threshold
Free demosaicing (GBTF) [18], and Integrated Gradients (IGD) [19]
methods. LPA, DLMMSE, and VCD are the highest performing
methods on the Kodak image set among eleven state-of-the-art demosaicing algorithms in [14]. RAD, GBTF, and IGD are recent methods
that are introduced after this survey paper.
The CPSNR results are summarized in Table I. The best result for
each image is highlighted with bold text. IGD method has the highest
average CPSNR with GBTF 0.02 dB behind it. The proposed algorithm
comes fourth very closely after LPA. In terms of individual images,

~
^
G = B + (W A )(G B ) + (1 W + A )
M2 ~
M3 ~
M1 ~
MTotal (G 2 B 2 ) + MTotal (G 2 B 2 ) + MTotal (G
A = min G 1 G +1 + G +1 G +1 + G +1 G 1 + G
4
0

i;j

i;j

i0 ;j 0

i;j

i;j 0

i;j

i0 ;j 0

i;j

i;j

i;j 0

i0 ;j

i;j

i;j

;j j

;j 0

~
B +2) + MM4 (G +2 B +2 )
Total
G 1 ;C2

i;j +2 0

i;j 0

i;j 0 j

i;j 01 0

i;j

i0 ;j j

;j 0

;j

(7)

396

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 21, NO. 1, JANUARY 2012

Fig. 2. Fence region from the lighthouse image (no. 8). (a) Original. (b) AP. (c) VCD. (d) DLMMSE. (e) LPA. (f) RAD. (g) GBTF. (h) IGD. (i) Proposed.

Fig. 3. Fence region from the lighthouse image (no. 8) corrupted with noise. (a) LPA. (b) GBTF. (c) IGD.

GBTF gives the best result on 5 out of 12 followed by the proposed

method with 3, and IGD and LPA methods with 2 each. Although the
proposed method does not have the highest CPSNR average, its results
are comparable to the latest demosaicing methods for the most part and
it outperforms all other methods on a number of images. As an alterna-

tive objective comparison measure, S-CIELAB results are summarized

in Table II.
The computational complexity of the proposed algorithm is higher
than VCD and IGD, in line with DLMMSE, and lower than POCS and
LPA methods. The proposed method requires 376 addition, 64 multi-

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 21, NO. 1, JANUARY 2012

plication, 52 absolute, 36 shift, and 10 division operations for every 2

by 2 GRBG input pixel block. The highest performing IGD method requires between 266 and 374 operations for the same 2 by 2 block [19].
A detailed complexity comparison table can be found in [19].
A challenging image region is presented in Fig. 2 for visual quality
comparison. The performance of the proposed solution under noise is
compared against three highest performing methods in Fig. 3. Note that
the LPA method has a noise reduction component built-in, so its result
has less visible noise but more false color artifacts. We observe that the
noise sensitivity of the proposed method is comparable with the GBTF
and IGD methods.

397

[17] D. Menon and G. Calvagno, Regularization approaches to demosaicking, IEEE Trans. Image Process., vol. 18, no. 10, pp. 22092220,
Oct. 2009.
[18] I. Pekkucuksen and Y. Altunbasak, Gradient based threshold free
color lter array interpolation, in Proc. IEEE Int. Conf. Image
Process., Sep. 2010, pp. 137140.
[19] K. H. Chung and Y. H. Chan, Low-complexity color demosaicing algorithm based on integrated gradients, J. Electron. Imag., vol. 19, no.
2, p. 021104-1-15, Jun. 2010.

Improved Image Recovery From Compressed Data

Contaminated With Impulsive Noise

IV. CONCLUSION
We presented a simple edge strength lter and applied it to the CFA
interpolation problem. The edge strength lter helped us identify the regions where constant color difference assumption is likely to fail which
in turn lead to improved demosaicing performance. Further research
efforts will focus on improving the interpolation results by exploiting
spectral correlation more effectively and applying the proposed edge
strength lter to other image processing problems.

REFERENCES
[1] B. E. Bayer, Color Imaging Array, U.S. 3 971 065, Jul. 1976.
[2] R. Kimmel, Demosaicing: Image reconstruction from color CCD samples, IEEE Trans. Image Process., vol. 8, no. 9, pp. 12211228, Sep.
1999.
[3] R. Lukac and K. N. Plataniotis, A normalized model for color-ratio
based demosaicking schemes, in Int. Conf. on Image Process., 2004,
vol. 3, pp. 16571660.
[4] J. F. Hamilton and J. E. Adams, Adaptive color plan interpolation in
single sensor color electronic camera, U.S. Patent 5 629 734, Mar. 13,
1997.
[5] C. A. Laroche and M. A. Prescott, Apparatus and method for adaptively interpolating a full color image utilizing chrominance gradients,
U.S. Patent 5 373 322, Dec. 13, 1994.
[6] J. W. Glotzbach, R. W. Schafer, and K. Illgner, A method of color
lter array interpolation with alias cancellation properties, in Proc.
IEEE Int. Conf. Image Process., 2001, vol. 1, pp. 141144.
[7] B. K. Gunturk, Y. Altunbasak, and R. M. Mersereau, Color plane interpolation using alternating projections, IEEE Trans. Image Process.,
vol. 11, no. 9, pp. 9971013, Sep. 2002.
[8] N.-X. Lian, L. Chang, Y.-P. Tan, and V. Zagorodnov, Adaptive ltering for color lter array demosaicking, IEEE Trans. Image Process.,
vol. 16, no. 10, pp. 25152525, Oct. 2007.
[9] R. H. Hibbard, Apparatus and method for adaptively interpolating a
full color image utilizing luminance gradients, U.S. Patent 5 382 976,
Jan. 17, 1995.
[10] J. E. Adams and J. F. Hamilton, Jr., Adaptive color plan interpolation
in single sensor color electronic camera, U.S. Patent 5 506 619, Apr.
9, 1996.
[11] K.-H. Chung and Y.-H. Chan, Color demosaicing using variance of
color differences, IEEE Trans. Image Process., vol. 15, no. 10, pp.
29442955, Oct. 2006.
[12] L. Zhang and X. Wu, Color demosaicking via directional linear minimum mean square-error estimation, IEEE Trans. Image Process., vol.
14, no. 12, pp. 21672178, Dec. 2005.
[13] D. Paliy, V. Katkovnik, R. Bilcu, S. Alenius, and K. Egiazarian, Spatially adaptive color lter array interpolation for noiseless and noisy
data, Int. J. Imag. Syst. Technol., vol. 17, no. 3, pp. 105122, 2007.
[14] X. Li, B. Gunturk, and L. Zhang, Image demosaicing: A systematic
survey, Proc. SPIEInt. Soc. Opt. Eng., vol. 6822, p. 68221J-1-15, Jan.
2008.
[15] K. Hirakawa and T. W. Parks, Adaptive homogeneity-directed demosaicing algorithm, IEEE Trans. Image Process., vol. 14, no. 3, pp.
360369, Mar. 2005.
[16] D. Menon, S. Andriani, and G. Calvagno, Demosaicing with directional ltering and a posteriori decision, IEEE Trans. Image Process.,
vol. 16, no. 1, pp. 132141, Jan. 2007.

Duc-Son Pham, Member, IEEE, and

Svetha Venkatesh, Senior Member, IEEE

AbstractCompressed sensing (CS) is a new information sampling

theory for acquiring sparse or compressible data with much fewer
measurements than those otherwise required by the Nyquist/Shannon
counterpart. This is particularly important for some imaging applications
such as magnetic resonance imaging or in astronomy. However, in the
norm on the residuals is not
existing CS formulation, the use of the
particularly efcient when the noise is impulsive. This could lead to
an increase in the upper bound of the recovery error. To address this
problem, we consider a robust formulation for CS to suppress outliers in
the residuals. We propose an iterative algorithm for solving the robust
CS problem that exploits the power of existing CS solvers. We also show
that the upper bound on the recovery error in the case of non-Gaussian
noise is reduced and then demonstrate the efcacy of the method through
numerical studies.
Index TermsCompressed sensing (CS), image compression, impulsive
noise, inverse problems, robust recovery, robust statistics.

I. INTRODUCTION
Compressed sensing (CS) [8], [12] is a new direct information sampling theory specically for the acquisition and recovery of sparse or
compressible data as an alternative to the existing Nyquist/Shannon
sampling counterpart to exploit the characteristics of the signals. This
theory is essentially a direct information sampling scheme. Such a
scheme is crucial for some applications where reducing the sensing
cost is desirable, such as MRI [25].
The CS theory has led to numerous computationally efcient recovery methods such as pursuit algorithms [11], [26], [28], optimization algorithms [16], [23], a complexity regularization algorithm [19],
and Bayesian methods [22]. The CS theory has been found useful in
a number of imaging applications, including MRI [25], astronomy [4],
and high-SNR image compression [18]. We refer the reader to the CS
repository (http://dsp.rice.edu/cs) for background material and current
development in this area. In the following, we assume that the reader is

Manuscript received March 02, 2010; revised August 26, 2010, December
02, 2010 and February 07, 2011; accepted July 11, 2011. Date of publication
September 12, 2011; date of current version December 16, 2011. The associate
editor coordinating the review of this manuscript and approving it for publication was Prof. Birsen Yazici.
The authors are with the Institute for Multisensor and Content Analysis,
Curtin University, Perth, WA 6845, Australia (e-mail: dspham@ieee.org;
svetha@cs.curtin.edu.au).
Color versions of one or more of the gures in this paper are available online
at http://ieeexplore.ieee.org.
Digital Object Identier 10.1109/TIP.2011.2162418

1057-7149/$26.00 2011 IEEE