Scientific Journal of Information Engineering

February 2014, Volume 4, Issue 1, PP.19-25

Elastic-Net Regression Algorithm Based on

Multi-Scale Gaussian Kernel
Yongli Xu#, Zhenjun Yang
Department of Mathematics, Beijing University of Chemical Technology, Beijing 100029, China
#

Email: xuyongli2312@sina.com; buct_yzj@126.com

Abstract
This paper proposes an elastic-net algorithm based on multi-scale Gaussian kernels to deal with the approximation of regression
function. We use Gaussian kernels with different kernel width to approximate the high and low frequency components of the
regression function; then weighted L1 norm and L2 norm of the prediction function utilized as regularization term. In the
simulation experiments, multiple Gaussian kernels based elastic-net obtains less prediction error and better sparse performance
than single Gaussian kernel based elastic-net. In addition, multiple Gaussian kernels based elastic-net can precisely predict the
high and low frequency components of objective function.
Keywords: Multi-kernel Based Elastic-net; Gaussian Kernel; Least Squares Regularized Regression; Sparsity

100029

L1
L2

Lasso (least solute shrinkage and selection operator)(ridge regression)

[2], [6]Lasso T-X

2
2

j
j 1

L1
0Lasso
Lasso L1 L2 Lasso
Lasso
Zou Hastie Lasso (elastic net)[2]
1 N

min p1 R (0 , ) min p1
(1)
( yi 0 xiT )2 P ( )
( 0 , )
( 0 , )
2
N
i 1

1
2
P ( ) (1 ) l l Lasso =0
2
1
2
=1 Lasso

11101024
- 19 http://www.sjie.org

[3]
[4]

[4]

[13]

k(x,z)=exp(- x z )
2 2

1
(X, d) Y=R Z=X*Y
f ( x) Y yd (y | x),

x X (y | x) x

m
z={(xi ,yi )}i
1

f z

f [11], [33]
m

f ( x) i K ( xi , x) ( 1 , ..., m )
i 1

min {

( 1 ,..., m )

1 m m
( i K ( xi , x j ) yi )2 P ( )}
m j 1 i 1

(2)

l K(x,z)=exp(- x z )
1
2 2
L1 L2

P ( ) (1 )

2
l2

L1 L2 0, L2
1, L1
LASSO
4 xi

(4)

i 1

i 1

f ( x) i K1 ( xi , x) mi K 2 ( xi , x)

K1 (x,z)=exp(-

xz

212

) K 2 (x,z)=exp(-

min {

( 1 ,..., 2 m )

xz
2 22

(3)

) ( 1 , ..., 2 m )

m
1 m m
( i K1 ( xi , x j )+ i m K2 ( xi , x j ))2 P ( )},
m j 1 i 1
i 1

- 20 http://www.sjie.org

(4)

 P ( ) (1 )

2
l2

l
1

L1 L2 0, L2
1, L1
LASSO

2.1
20 3N
f

etrain
etest

3N

(yi y i ) / N evalidation
i 1
2

2N

i N 1

(yi y i ) / N

(yi y i ) / N y y

i 2 N 1

20
12 1 2 2 1 2
122
1 1 2 2
f 1 2 f 1 2

2 f
MATLAB
R2007b 2.5GHzCPU

2.2

y=5sin(2x)+sin(30x), 0 x .

{( xi , f1* (xi ) t i )}iN1 xi (0, ) i [0.5,0.5]

t {0.1,0.5} a (i 101 )10
i 0 {10,20,30,40,50,60}
2 10
1 , 2 {108 +i*10-6 }100
i 1 1 , 2 {i 10 }i 1
2 14
1 2 1 , 2 {108 +i*10-6 }100
i 1 {i*10 }i 1 1

2 t 0.5 00.51
L1 3 4 t 0.1 00.51
L1
1 t 0.5 a 00.51 N 10,20,30,40,50,60

- 21 http://www.sjie.org


=0
=0
=0.5
=0.5
=1
=1

10
2.36
2.40
2.31
2.41
2.31
2.42

20
2.06
1.79
2.06
1.76
2.05
1.75

30
1.34
1.09
1.31
1.11
1.29
1.13

40
1.14
0.90
1.13
0.92
1.12
0.91

50
0.75
0.61
0.73
0.61
0.74
0.59

60
0.50
0.43
0.51
0.43
0.51
0.42

2 L1 t 0.5 a 00.51 N 10,20,30,40,50,60

=0
=0
=0.5
=0.5
=1
=1

10
41.05
34.92
35.28
34.08
35.03
33.29

20
104.21
84.44
96.91
78.46
95.08
70.76

30
185.60
121.49
181.33
122.97
180.72
130.98

40
309.18
171.27
299.91
159.04
284.74
152.75

50
287.50
251.57
278.33
240.53
262.27
179.60

60
212.71
200.18
190.53
185.88
177.92
169.89

3 t 0.1 a 00.51 N 10,20,30,40,50,60

=0
=0
=0.5
=0.5
=1
=1

10
2.43
2.38
2.50
2.38
2.46
2.40

20
1.94
1.56
1.95
1.55
1.92
1.58

30
1.53
1.26
1.49
1.27
1.47
1.27

40
0.89
0.79
0.88
0.80
0.90
0.81

50
0.69
0.56
0.70
0.56
0.70
0.56

60
0.51
0.42
0.51
0.42
0.51
0.43

4 L1 t 0.1 a 00.51 N 10,20,30,40,50,60

=0
=0
=0.5
=0.5
=1
=1

10
45.07
37.71
44.82
37.04
44.10
36.06

20
110.26
62.79
109.92
60.96
165.17
58.70

30
244.77
121.93
226.14
113.90
213.65
110.73

40
255.64
145.16
247.58
132.05
248.36
124.02

50
358.83
181.83
349.86
166.34
311.82
157.27

60
307.32
176.78
293.54
165.09
288.85
149.17

2
0

-2

-2

-4

-4

-6
0

0.5

1.5

2.5

-6
3.5 0

0.5

(a)

1.5

2.5

3.5

1 N 45

1 3

2 4
L1
- 22 http://www.sjie.org


L1 L1
1 45 0.5 0.5

2 200
2 =0
=1
=0.4
4

6
4

2
0

-2
-2

-4
-6
0

0.5

1.5

2.5

-4

3.5 0

0.5

a 0

1.5

2.5

3.5

b 0

6
4

1
2
0

-2

-1
-4
-6
0

0.5

1.5

2.5

c 0.4

-2
3.5 0

0.5

1.5

2.5

3.5

1.5

2.5

3.5

d 0.4

6
4

1
2

0
-2

-1
-4
-6
0

0.5

1.5

2.5

e 1

3.5

-2
0

0.5

f 1

2 N 250 t 0.1

2.3
600
3
- 23 http://www.sjie.org

 20

i 5
1 , 2 {1010 +i*10-6 }100
i 1 1 , 2 {2 }i 1
i 5
1 , 2 {1010 +i*10-6 }100
i 1 {2 }i 1 5

L1 5

L1
L1
20
18

y m/s

16
14
12
10
8
6
4
2
0

100

200

300

400

500

600

x
3
5 L1
a
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1

1.386
1.386
1.386
1.386
1.386
1.386
1.387
1.387
1.387
1.387
1.388

1.289
1.288
1.287
1.286
1.286
1.286
1.285
1.284
1.284
1.284
1.283

L1
3092
3051
3010
2960
2913
2873
2826
2776
2726
2679
2635

L1
452
449
446
443
440
437
434
432
430
427
425

loss

REFERENCES
[1]

Yong-Li Xu, Di-Rong Chen, Han-Xiong Li. Least Square Regularized in Sum Space[J], IEEE TRANSACTION NETWORKS AND
- 24 http://www.sjie.org

LEARNING SYSTEMS, VOL, NO.4, APRIL 2013: 635

[2]

De Mol, C.De Vito, E.Rosasco, L,et al. Elastic-Net Regularization in Learning Theory[J].2008,.DOI:

[3]

Kui Xiang,Bing-nan Li. Sparsity in Principal Component Analysis:A Survey[J]. Acta Electronica Sinica, 2012, 40(12):2525-2532.
DOI:10.3969/ j.issn.0372-2112.2012.12.027

[4]

Hong-Qiao Wang, Yan-Ning Cai, Fu-Chun Sun, Zong-Tao Zhao. Adaptive Sequence Learning and Applications for Multi-Scale
Kernel Method [J]. PATTERN RECOGNITION AND ARTIFICIAL INTELLIGENCE, YR 2011, VO 24, IS 1: OP 72-81

[5]

Yan-Jun, Jue Wang. A Bi-Sparse Relational Learning Algorithm Based on Multiple Kernel Learning[J]. JOURNAL OF
COMPUTER RESEARCH AND DEVELOPMENT,YR 2010, VO 47, IS 8, OP1400

[6]

.[D].,2006.DOI:10.7666/d.d047298

[7]

.[D].,2011.DOI:10.7666/d.y1918551

[8]

Chun-Ying Liu, Yan-Xiong Fang, Yu-Mian Yu. Research into Simultaneous Determination of Five Components by Ridge
Regression Spectrophotometry[J].SPECTROSCOPY AND SPECTRAL ANALYSIS, 1999,19(4):629-631

[9]

Jerome Friedman,Trevor Hastie,Rob Tibshirani,et al. Regularization Paths for Generalized Linear Models via Coordinate Descent
[J]. Journal of Statistical Software,2010,33(01)

1982-

1991-

Email: buct_yzj@126.com

Email: xuyongli2312@sina.com

- 25 http://www.sjie.org