2011.5.

20

CDN


Alexa137
20000

100%

99.999%

CDN

CDN
Global Load Balancing

L1-Cache

L2-Cache

Application

Storage

Cache
Cluster

Cache
Cluster

Cache
Cluster

Cache
Cluster

Cache
Cluster

Image Server
Cluster

Cache
Cluster

Cache
Cluster

Image Server
Cluster

~40T

~80T

~200 Servers
2000T Used /

TFS
Cluster

TFS
Cluster

3000T Deployed
6

2007
Upload Server

Admin Server

Image Server

SnapVault

SnapMirror

SnapMirror

NearStore R200
DR: A+B+C+D
SnapShot

FAS980C

SnapShot

SnapShot

FAS980C
C: Online Image
D: Online Image

A: Online Image
B: Online Image

IDC

SnapShot

2006
7
NetApp 980C/R200
20TB

10T

8

TFS 1.0
20076

TFSTaobao File System1.0

200PC Server(146G*6 SAS 15K Raid5)

:
: 140 TB
: 50 TB
IOPS 200+3MBps
9

TFS1.0

Name ServerData Server

Data ServerLinux
block(64M
block)
TFS

T2auNFXXBaXXXXXXXX_!!140680281.jpg
block_noobject_no

block
ext3
raid5
10

TFS 1.3
20096
TFSTaobao File System1.3

2010.8.22
440PC Server (300G*12 SAS 15K RPM) + 30PC
Server (600G*12 SAS 15K RPM)

1800 TB
995TB
Data ServerIOPS 900+15MB+
Name Server217MB

2010PC Server (1T SATA * 12)

11

TFS1.3
TFS1.3





Name Server

12

CDN
CDN2011.5
500T + 500T
57317.45K
8K61%11%

CDN
55
10G
CDN550G
318G
13

CDN-
client
VIP1

VIP2

NetScaler(7LB)

squid

squid

NetScaler(7LB)

squid

squid

14

CDN-
client
VIP1

LVS(L4)
Haproxy(L7)

VIP2

LVS (L4)

haproxy(L7)

Haproxy(L7)

squid

squid

squid

squid

15

CDN
\

128squidsquid
2%
VIP100G

Hash
1/(n+1)

Squid
COSSTCOSSFIFO
1T
SquidSquid
1250Mmemory
cache
sendfilepage
cache
SSDDIRECT_IO
SAS/SATApage cache
SquidSSD+SAS+SATA
GDSF

17

1
SSDSAS
SATA
SSD + 4 * SAS + SATA
[root@cache161 ~]# iostat -x -k 60 | egrep -v -e sd.[1-9]

avg-cpu: %user %nice %system %iowait %steal %idle

3.15 0.00 5.63 11.35 0.00 79.87
Device:
sda
sdb
sdc
sdd
sde
sdf

rrqm/s
15.40
0.07
5.73
5.57
5.53
5.45

wrqm/s r/s w/s rkB/s

1.17 50.66 2.63 2673.22
0.03 447.29 1.02 4359.01
1.53 114.93 8.42 1264.86
2.07 121.83 9.57 1319.45
1.45 111.45 8.52 1246.53
2.02 118.93 8.00 1281.92

wkB/s avgrq-sz avgqu-sz await svctm %util

124.85 105.01
0.55 10.39 6.27 33.41
191.90 20.30
0.32 0.71 0.27 12.13
100.58 22.14
1.05 8.48 3.56 43.94
104.12 21.67
1.19 9.02 3.63 47.72
101.92 22.48
0.95 7.88 3.42 41.06
106.25 21.87
1.19 9.37 3.74 47.44

SATASSDSAS
4SASSSD
18

2
SSD
SASSATA
SSD + 4 * SAS + SATA
[root@cache161 ~]# iostat -x -k 60 | egrep -v -e sd.[1-9]

avg-cpu: %user %nice %system %iowait %steal %idle

3.15 0.00 5.63 11.35 0.00 79.87
Device:
sda
sdb
sdc
sdd
sde
sdf

rrqm/s
5.08
1.68
0.22
0.23
0.18
0.10

wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
1.65 18.55 2.52 1210.07 119.00 126.18
0.14 6.50 5.46 11.51
0.05 610.53 1.75 6962.29 413.47 24.09
0.28 0.46 0.23 14.25
0.03 28.87 0.97 1172.93 189.13 91.31
0.16 5.28 4.40 13.13
0.02 29.70 0.77 1133.47 122.53 82.45
0.15 4.99 4.39 13.37
0.03 28.23 1.03 1078.73 206.27 87.81
0.15 5.00 4.24 12.40
0.02 28.42 0.55 1090.27 115.00 83.22
0.15 5.04 4.44 12.86

SATASSDSAS
SSD4SAS5SASSATA

19

32 DELL R710
2 LVS + 32 Haproxy + 64 Squid
2009.12.21
16Gbps
R710500Mbps
squidobject1800
Cache97%
Cache97%

20

30 DELL PowerEdge 2950

2 LVS + 30 Haproxy + 60 Squid
20105
12Gbps
2950400Mbps
160G SSD + 143G SAS * 4 + 1T SATA
squidobject3000
Cache97.5%
Cache97.5%

21

22

CDN

23

24

TFS

25

TFS
TFS




+EXT3/EXT4

TFS

3P

TFS


26

TAIR

27

TAIR
TAIR
DHTKV



Json
MEM/BDB/EXT3/EXT4

TAIR

T/

TAIR
CAP

28

OceanBase

201040
60

OceanBase

110ms
1200QPS/
2011.5
SSD
29

MySQL
MySQL

MySQL
PCI-E Flashcache

Innodb IO

MySQL
TDDL, JDBC

30



Apache/Nginx
HTTP
Squid

LVS
HAProxy

TCP
31

Java
Java
Java
JavaHotspots VMOpen JDK VM
Linux
HypervisorJava without OS

CPUJVM
Intel CPUJVM
JVMGC Invisible Heap
32

Linux

Linux

Linux

Linux

33

34



CPUIntel ATOM, VIA Nano
ChipsetSSDSATA
GPUUSB Controller

CPUI/O
CDN Cache Servermemory cache
Web Server

()I/O

35

CDN:

LVS

LVS

Rack

Internet

6Gbps

CDN
CDN
64
CPU4G
80G SSD500G SATA
25W
LVS

1100QPS
10G
CPU
37

CDN

CDN
CDN
CDN

CDN
CDN

CDNBGP
38

CDN
CDN

GTM
CDNCDN

CDN

39

CDN

40

41


Code.taobao.org

Key/Value Cache/Store TAIR
TFS
WebWebx
TaobaoABS

42

CDN

43

--

44

Q&A
!

45