DDoS protection

Using Netfilter/iptables

Jesper Dangaard Brouer

Senior Kernel Engineer, Red Hat Network-Services-Team Dev on!"c# $e% &'()

1/36

Email* %rouer+red,at"com - netoptimi#er+%rouer"com - ,awk+kernel"org DDoS protection using Netfilter/iptables

.,o am /

Name* Jesper Dangaard Brouer

0inu1 Kernel Developer at Red Hat Edu*

omputer Science !or 2ni"

open,agen

$ocus on Network, Dist" s3s and 4S S3sadm, Kernel Developer, Em%edded

0inu1 user since (556, pro!essional since (557

4penSource pro8ects, aut,or o!

9DS0-optimi#er, :9N /:Ta%les**li%iptc, /:T;-9nal3#er 0inu1 kernel, iproute&, ipta%les, li%pcap and .ires,ark

:atc,es accepted into

4rgani#er o! Net!ilter .orks,op &'(<

DDoS protection using Netfilter/iptables

2/36

.,at will 3ou learn=

0inu1 Kernel is vulnera%le to simple S>N attacks End-,ost mitigation?s alread3 implemented in kernel

s,ow it is not enoug, solution is stalled """ ,ow to work-around t,is

Kernel* serious @listen@ socket scala%ilit3 pro%lem

$irewall-%ased solution* s3npro13 Aipta%les-net!ilterB How !ast is state!ul !irewalling

.,ere is our pain points 0earn Net!ilter tricks* %oost per!ormance a !actor ('
DDoS protection using Netfilter/iptables

3/36

$irst* Basic N/

tuning ('(

9ll tests in presentation Basic tuning

$irst kill CirD%alanceE N/ ,ardware Dueue, are :2 aligned

Disa%le Et,ernet !low-control

/ntel i1g%e ,w-driver issue

single %locked ,w Dueue %locks ot,ers

$i1 in kernel v<"F"' commit <e%e7!de%' Ai1g%e* Set DropGEN %it w,en multiple R1 Dueues are present w-o !low controlB

4/36

DDoS protection using Netfilter/iptables

$ocus* $looding DoS attack

Denial o! Service ADoSB attacks $ocus* T : !looding attacks

9ttacking t,e <-Wa3 HandS,ake A<.HSB End-,ost resource attack

S>N !lood S>N-9 K !loods 9 K !loods A<rd packet in <.HSB

9ttacker o!ten spoo!s src /:

T : S>N $looding 9ttacks and ommon Iitigations

Descri%ed in R$ )57H*

5/36

DDoS protection using Netfilter/iptables

0inu1 current end-,ost mitigations

Jargon R$ )57H AT : S>N $looding 9ttacks and ommon IitigationsB

0inu1 uses ,3%rid solution

S>N Ccac,eE Iini reDuest socket Iinimi#e state, dela3 !ull state alloc S>N C%acklogE o! outstanding reDuest sockets

9%ove limit, use S>N CcookiesE

6/36

DDoS protection using Netfilter/iptables

Details* S>N @cac,e@ savings

Small initial T B ATransmission ontrol BlockB struct reDuestGsock Asi#e F6 %3tesB

mini sock to represent a connection reDuest S09B %e,ind ,ave si#eo!Astruct tcpGreDuestGsockB
Structs em%edded in eac,-ot,er

But alloc si#e is ((& %3tes

F6 %3tes JJ struct reDuestGsock 7' %3tes JJ struct inetGreDuestGsock ((& %3tes JJ struct tcpGreDuestGsock

$ull T B Astruct inetGsockB is 7<& %3tes

Anote, si#es will increase-c,ange in more recent kernelsB

7/36

DDoS protection using Netfilter/iptables

Details* /ncreasing S>N %acklog

Not recommended to increase !or DoS

4nl3 increase, i! legitimate tra!!ic cause log*

CT :* :ossi%le S>N !looding """E

/ncreasing S>N %acklog is not o%vious

9d8ust all t,ese*

-proc-s3s-net-ipv)-tcpGma1Gs3nG%acklog -proc-s3s-net-core-soma1conn S3scall listenAint sock!d, int backlogBK

8/36

DDoS protection using Netfilter/iptables

S>N cookies

Simpli!ied description

S>N packet

don?t create an3 local state Encode state in SELM Aand T : optionsB ontains SELMN( Aand T : timestampB Recover state ;alidate A<.HSB 9 K packet state

S>N-9 K packet

9 K packet

SH9 ,as, is computed wit, local secret

9/36

DDoS protection using Netfilter/iptables

Details* S>N-cookies

S>N cookies SH9 calculation is e1pensive SNI: counters ASince kernel v<"(B

TCPReqQFullDoCookies * num%er o! times a S>N 44K/E was replied to client TCPReqQFullDrop * num%er o! times a S>N reDuest was dropped %ecause s3ncookies were not ena%led" -proc-s3s-net-ipv)-tcpGs3ncookies J &

9lwa3s on option

10/36

DDoS protection using Netfilter/iptables

So, w,at is t,e pro%lem=

Oood End-Host counter-measurements :ro%lem* 0/STEN state scala%ilit3 pro%lem

;ulnera%le !or all !loods

S>N, S>N-9 K and 9 K !loods

Num%ers* Peon :2 PFFF' ('O i1g%e

N4 0/STEN socket*

&"5')"(&7 pkts-sec -- S>N attack &F&"'<& pkts-sec -- S>N attack <<6"FH6 pkts-sec -- S>NN9 K attack <<("'H& pkts-sec -- 9 K attack
DDoS protection using Netfilter/iptables

0/STEN socket*

11/36

:ro%lem* S>N-cookie vs 0/STEN lock

Iain pro%lem*

S>N cookies live under 0/STEN lock

,ttp*--t,read"gmane"org-gmane"linu1"network-&<&&<7

/ proposed S>N %rownies !i1 AIa3 &'(&B

Oot re8ected, %ecause not general solution

e"g" don?t ,andle S>N-9 K and <.HS

Need to C!orward-portE patc,es
ABug ('FH<6) - R$E* :arallel S>N cookies ,andlingB

N$.S&'(< got clearance as a !irst step solution

12/36

DDoS protection using Netfilter/iptables

$irewall and :ro13 solutions

Net ork!"ase# ountermeasures

.esle3 I" Edd3, descri%es S>N-pro13

/n isco* T,e /nternet :rotocol Journal - ;olume 5, Num%er ), &''6, link* ,ttp*--goo"gl-9 (99Q 9vail in kernel <"(< and RHE0H

Net!ilter* ipta%les target S$NPR%&$

B3 :atrick IcHard3, Iartin Top,olm and Ie

9lso works on local,ost Oeneral solution Solves S>N and 9 K !loods /ndirect trick also solves S>NN9 K

DDoS protection using Netfilter/iptables

13/36

S>N pro13 concept

14/36

DDoS protection using Netfilter/iptables

onntrack per!ormanceA(B

S>N:R4P> needs conntrack

.ill t,at %e a per!ormance issue=

&"56)"'5( pkts-sec -- N4 0/STEN sock N no ipta%les rules &))"(&5 pkts-sec -- 0/STEN sock N no ipta%les rules )<F"F&' pkts-sec -- N4 0/STEN sock ' conntrack (H&"55& pkts-sec -- 0/STEN sock ' conntrack %ut / ,ave some tricks !or 3ou K-B

Base per!ormance*

0oading conntrack* AS>N !lood, causing new conntrackB

0ooks %ad"""

15/36

DDoS protection using Netfilter/iptables

onntrack per!ormanceA&B

onntrack Alock-lessB lookups are really fast

:ro%lem is insert and delete conntracks 2se to protect against S>NN9 K and 9 K attacks 9llow 9 K pkts to create new connection Disa%le via cmd*
sysctl -w net/netfilter/nf_conntrack_tcp_loose=0

De!ault net!ilter is in T : ClooseE mode

Take advantage o! state C/N;90/DE

Drop invalid pkts before reac,ing 0/STEN socket

iptables -m state --state INVALID -j DR !

16/36

DDoS protection using Netfilter/iptables

onntrack per!A<B 9 K-attacks

(C) attacks, conntrack per!ormance De!ault ClooseJ(E and pass /N;90/D pkts

(H5"'&H pkts-sec &<F"5') pkts-sec Alisten lock scalingB F"F<<"'F6 pkts-sec

0ooseJ' and and pass /N;90/D pkts

0ooseJ' and and DR4: /N;90/D pkts

17/36

DDoS protection using Netfilter/iptables

onntrack per!A)B S>N-9 K attack

S$N!(C) attacks, conntrack per!ormance

S>N-9 Ks don?t auto create connections T,us, c,anging ClooseE setting is not important &<'"<)7 pkts-sec F"<7&"&6F pkts-sec F")'7"<'H pkts-sec

De!ault pass /N;90/D pkts Aand ClooseJ(EB

De!ault DR4: /N;90/D pkts Aand ClooseJ(EB

De!ault DR4: /N;90/D pkts Aand ClooseJ'EB

18/36

DDoS protection using Netfilter/iptables

S3npro13 per!ormance

%nl* conntrack S$N attack proble+ left

Due to conntrack insert lock scaling &))"(&5 pkts-sec -- 0/STEN sock N no ipta%les rules (H&"55& pkts-sec -- 0/STEN sock ' conntrack

Base per!ormance*

0oading conntrack* AS>N !lood, causing new conntrackB

Using S$NPR%&$

,-./0-.,1 pkts-sec -- 0/STEN sock N s*npro2* N conntrack

19/36

DDoS protection using Netfilter/iptables

ipta%les* s3npro13 setupA(B

2sing S>N:R4P> target is complicated

S>N:R4P> works on untracked conntracks

/n CrawE ta%le, CnotrackE S>N packets*

iptables -t raw -I PREROUTING -i $DEV -p tcp -m tcp --syn \ --dport $PORT -j CT --notrack

20/36

DDoS protection using Netfilter/iptables

ipta%les* s3npro13 setupA&B

Iore strict conntrack ,andling

Need to get unknown 9 Ks A!rom <.HSB to %e marked as /N;90/D state

Aelse a conntrack is 8ust createdB

Done %3 s3sctl setting*

sbin s!sctl -" net net#ilter n#$conntrack$tcp$loose%&

21/36

DDoS protection using Netfilter/iptables

ipta%les* s3npro13 setupA<B

atc,ing state*

2NTR9 KED JJ S>N packets /N;90/D JJ 9 K !rom <.HS

2sing S>N:R4P> target*

iptables -' INPUT -i $DEV -p tcp -m tcp --dport $PORT ( -m state --state INV')ID*UNTR'C+ED ( -j SYNPROXY --sack-perm --timestamp --"scale , --mss -./&

22/36

DDoS protection using Netfilter/iptables

ipta%les* s3npro13 setupA)B

Trick to catc, S>N-9 K !loods

Drop rest o! state /N;90/D, contains S>N-9 K

iptables -' INPUT -i $DEV -p tcp -m tcp --dport $PORT ( -m state --state INV')ID -j DROP

Ena%le T : timestamping

Because S>N cookies uses T : options !ield

sbin s!sctl -" net ip0. tcp$timestamps%-

23/36

DDoS protection using Netfilter/iptables

ipta%les* s3npro13 setupAFB

onntrack entries tuning

Ia1 possi%le entries & Iill

&77 %3tes R & Iill J FH6"' IB

net net#ilter n#$conntrack$ma1%2&&&&&&

/I:4RT9NT* 9lso ad8ust ,as, %ucket si#e

-proc-s3s-net-net!ilter-n!GconntrackG%uckets writea%le via -s3s-module-n!Gconntrack-parameters-,as,si#e

Has, 7 %3tes R &Iill J (6 IB ec3o 2&&&&&& 4 s!s mod5le n#$conntrack parameters 3as3si6e

24/36

DDoS protection using Netfilter/iptables

:er!ormance S>N:R4P>

Script ipta%lesGs3npro13"s, avail ,ere*

,ttps*--git,u%"com-netoptimi#er-network-testing-%lo%-master-ipta%les-ip ta%lesGs3npro13"s,

2sing S>N:R4P> under attack t3pes*

&"765"7&) pkts-sec S S>N-!lood )"5)7")7' pkts-sec S 9 K-!lood F"6F<"(&' pkts-sec S S>NN9 K-!lood

25/36

DDoS protection using Netfilter/iptables

S>N:R4P> parameters

T,e parameters given to S>N:R4P> target

Iust matc, t,e %ackend-server T : options Ianual setup A,elper tool n!s3npro13B 4nl3 one setting per rule Not use!ul !or DH : %ased network 9uto detect server T : options Simpl3 allow !irst S>N t,roug,

Future plan

atc, S>N-9 K and decode options

ARHBQ ('F56H5 - R$E* S3npro13* auto detect T : optionsB

26/36

DDoS protection using Netfilter/iptables

Real-li!eA(B* Handle 5'' Kpps

27/36

DDoS protection using Netfilter/iptables

Real-li!eA&B* SH9 sum e1pensive

S>N cookie SH9 sum is e1pensive

Bug ('FH<F& - R$E* /mprove S>N cookies calculations

28/36

DDoS protection using Netfilter/iptables

Real-li!eA<B* 4ut tra!!ic normal

29/36

DDoS protection using Netfilter/iptables

/ssue* $ull connection scala%ilit3

Still e1ists* Scala%ilit3 issue wit, !ull conn

Iade it signi!icantl3 more e1pensive !or attackers

At,e3 need real ,ostsB

$uture work* !i1 scala%ilit3 !or

entral lock* 0/STEN socket lock entral lock* Net!ilter new conntracks A.ork-in-progressB

30/36

DDoS protection using Netfilter/iptables

$i1ing central conntrack lock

onntrack issue

/nsert - delete conntracks takes central lock .orking on removing t,is central lock

ABased on patc, !rom Eric Duma#etB

ARHBQ (')<'(& - @net!ilter* conntrack* remove t,e central spinlock@B

:reliminar3 results, S>N-!lood

No 0/STEN socket to leave out t,at issue

)<F"F&' pkts-sec S conntrack wit, central lock ("6&6"H76 pkts-sec S conntrack wit, parallel lock

31/36

DDoS protection using Netfilter/iptables

Hack* Iulti listen sockets

Hack to work-around 0/STEN socket lock

Simpl3 0/STEN on several ports 2se ipta%les to rewrite-DN9T to t,ese ports

32/36

DDoS protection using Netfilter/iptables

Hack* $ull conn ,as,limit trickA(B

:ro%lem* $ull connections still ,ave scala%ilit3 :artition /nternet in -&) su%nets

A(&7R&F6R&F6 - &'5H(F& J ) ma1 ,as, listB

0imit S>N packets e"g" &'' S>N pps per src su%net Iem usage* !airl3 ,ig,

$i1ed* ,ta%le-si#e &'5H(F& R 7 %3tes J (6"H IB ;aria%le* entr3 si#e (') %3tes R F''''' J F& IB

33/36

DDoS protection using Netfilter/iptables

Hack* $ull conn ,as,limit trickA&B

2sing ,as,limit as work-around

9ttacker needs man3 real ,osts, to reac, !ull conn scala%ilit3 limit

iptables -t ra" -' PREROUTING -i $DEV ( -p tcp -m tcp --dport 7& --s!n ( -m 3as3limit ( --3as3limit-abo0e 2&& sec --3as3limit-b5rst -&&& ( --3as3limit-mode srcip --3as3limit-name s!n ( --3as3limit-3table-si6e 2&8,-92 ( --3as3limit-srcmask 2. -j DROP

34/36

DDoS protection using Netfilter/iptables

9lternative usage o! @socket@ module

9void using conntrack

2se 1tGsocket module

$or local socket matc,ing an !ilter out <.HS-9 Ks Aand ot,er com%osB

:arameter --nowildcard :ro%lem can still %e invalid-!lood 9 Ks Iitigate %3 limiting e"g",as,limit

Didn?t scale as well as e1pected

,ttps*--git,u%"com-netoptimi#er-network-testing-%lo%-master-ipta%les-ipta%lesGloc alGsocketG,ack"s,

35/36

DDoS protection using Netfilter/iptables

T,e End

T,anks to Iartin Top,olm and 4ne"com

$or providing real-li!e attack data

,ttp*--people"net!ilter"org-,awk-presentations-devcon!&'()-

Download slides ,ere*

$eed%ack-rating o! talk on*

,ttp*--devcon!"c#-!-<H Luestions=

/! unlikel3Atime !or DuestionsB

36/36

DDoS protection using Netfilter/iptables

E1tra Slides

37/36

DDoS protection using Netfilter/iptables

Disa%le ,elper auto loading

De!ault is to auto load conntrack ,elpers

/t is a securit3 riskT

:oking ,oles in 3our !irewallT

Disa%le via cmd*

ec"o 0 # /proc/sys/net/netfilter/nf_conntrack_"elper

ontrolled con!ig e1ample*

iptables -t raw -p tcp -p $%$% -j &' --"elper ftp

Read guide ,ere*

,ttps*--,ome"regit"org-net!ilter-en-secure-use-o!-,elpers-

38/36

DDoS protection using Netfilter/iptables