Optimizing Linux Servers

Davor Guttierrez
dguttierrez@me.com
3 Gen d.o.o.

Davor Guttierrez
3 Gen d.o.o
since 1996
located in Ljubljana; Slovenia
33 emploees
large !"# main$rame sstems and
%&!'(oriented sstems along )it* databases
more on ))).3Gen.si or $aceboo+.com,3GenS!

Agenda

-*at is optimization.

/er$ormance 0so$t)are , *ard)are1

Server optimization

/er$ormance monitoring

Sstem #onitoring 2ools

"enc*mar+ing 2ools

What is optimization?

3ur server is slo)

)e *ave ne) ver e4pensive server but ...

)e *ave ne) Linu4 distribution but ...

-*at is slo) in our server.

too man services running5 ...

dis+5 !,35 net)or+ con$iguration5 ...

Performance
To boost performance of a server, we
need just both its hardware and
software components to make it
operate efficiently.

Server optimization
Optimization can include fine tuning of

web servers (Apache, lighttpd, nginx etc),

disk !", block devices, #A$ or different filesystems (including %&% and %%$ devices),

kernel,

network !", T&'!' network stack,

firewall etc.
as well as

databases optimi(ation ) benchmarking and profiling, finding bottlenecks,

settings optimi(ation*

data storages tuning*

disk and memory usage optimi(ation

Start here with installation

6l)as ma+e custom installation o$ server5 don7t use de$ault

settings and de$ault install

8ustom partitioning5multiple $ile sstems5 ...

!nstall onl needed pac+ages5 never install )*ole group o$

pac+ages 9 ma+e minimal installation and t*en add
pac+ages

:ou don7t need ' -indo) and G&3#; on our server or

ou do.

8onsole is <=4>? c*aracters long and not 1=>@4A6<

Performance monitoring

Linu4 sstem administrators s*ould be pro$icient in Linu4

per$ormance monitoring and tuning.2o identi$ sstem
bottlenec+s and come up )it* solutions to $i4 it5 ou s*ould
understand *o) various components o$ Linu4 )or+s.

3n a ver *ig* level5 $ollo)ing are t*e $our subsstems t*at

needs to be monitoredB

8/%

&et)or+

!,3

#emor

CPU

:ou s*ould understand t*e $our critical per$ormance

metrics $or 8/%

conte4t s)itc*5

run Cueue5

cpu utilization5

load average.

Context Switch

-*en 8/% s)itc*es $rom one process 0or t*read1 to anot*er5 it

is called as conte4t s)itc*

-*en a process s)itc* *appens5 +ernel stores t*e current

state o$ t*e 8/% 0o$ a process or t*read1 in t*e memor

Dernel also retrieves t*e previousl stored state 0o$ a process

or t*read1 $rom t*e memor and puts it in t*e 8/%

8onte4t s)itc*ing is ver essential $or multitas+ing o$ t*e 8/%

6 *ig*er level o$ conte4t s)itc*ing can cause per$ormance

issues

Run queue

Eun Cueue indicates t*e total number o$ active

processes in t*e current Cueue $or 8/%

-*en 8/% is read to e4ecute a process5 it pic+s it up

$rom t*e run Cueue based on t*e priorit o$ t*e process

/rocesses t*at are in sleep state5 or !,3 )ait state are

not in t*e run Cueue

6 *ig*er number o$ processes in t*e run Cueue can

cause per$ormance issues

CPU Utilization

2*is indicates *o) muc* o$ t*e 8/% is currentl

getting used

2*is is $airl straig*t $or)ard5 and ou can vie) t*e

8/% utilization $rom t*e top command

1==F 8/% utilization means t*e sstem is $ull

loaded

6 *ig*er F o$ 8/% utilization )ill cause per$ormance

issues

LOAD average

2*is indicates t*e average 8/% load over a speci$ic time period.

3n Linu45 load average is displaed $or t*e last 1 minute5 ? minutes5

and 1? minutes. 2*is is *elp$ul to see )*et*er t*e overall load on t*e
sstem is going up or do)n.

Load average o$ G=.>? 1.>= 1.9=H indicates t*at t*e load on t*e
sstem is coming do)n. =.>? is t*e load average in t*e last 1 minute.
1.>= is t*e load average in t*e last ? minutes. 1.9= is t*e load
average in t*e last 1? minutes.

2*is load average is calculated b combining bot* t*e total number

o$ process in t*e Cueue5 and t*e total number o$ processes in t*e
uninterruptable tas+ status.

Disk I/O optimization

Linu4 currentl s*ips )it* $our di$$erent !,3 sc*edulers. 2*e areB deadline5 noop5
anticipator5 and c$C. 2*ere are man di$$erences bet)een t*ese sc*eduling
algorit*msB

8IJB 2*is is t*e de$ault algorit*m in most Linu4 distributions. !t attempts to distribute
all !,3 band)idt* evenl among all processes reCuesting !,3. !t is ideal $or most
purposes.

&33/B 2*e noop algorit*m attempts to use as little cpu as possible. !t acts as a basic
I!I3 Cueue e4pecting t*e *ard)are controller to *andle t*e per$ormance operations
o$ t*e reCuests.

6nticipatorB 2*is algorit*m attempts to reorder all dis+ !,3 operations to optimize dis+
see+s. !t is designed to increase per$ormance on sstems t*at *ave slo) dis+s.

DeadlineB 2*is sc*eduling algorit*m places !,3 reCuests in a priorit Cueue so eac* is
guaranteed to be ran )it*in a certain time. !t is o$ten used in real(time operating
sstems.

System scheduler how to change

cat /sys/block/sda/queue/scheduler
c*ange it )it*B

echo noop > /sys/block/sda/queue/scheduler

in SuS; Linu4 via :aS2 ...

More about System scheduler

8*anging sc*edulers on t*e $l allo)s ou to test and

benc*mar+ t*e algorit*ms $or our speci$ic application

3nce t*e c*ange is issued5 an current !,3 operations )ill

be e4ecuted be$ore t*e ne) sc*eduler goes into e$$ect5 so
t*e c*ange )ill not be instant

6lso remember t*at once one is set and per$orms to our

li+ing5 be sure to set t*e c*ange to be applied on
subseCuent reboots

And a little bit more ...

!t is o$ten recommend to use noop or deadline on an

SSD drive

2*ere is usuall no de$initive ans)er to )*ic* algorit*m to

use

"enc*mar+ing eac* one )ill be our best option

2*ere are cases )*ere c$C ma not be t*e best sc*eduler

$or our sstem or application

6n e4ample is i$ ou are running a E6!D dis+ arra )it* a

cac*ing raid controller

I/O optimization

!,3 )ait is t*e amount o$ time 8/% is )aiting $or !,3. !$ ou

see consistent *ig* i,o )ait on ou sstem5 it indicates a
problem in t*e dis+ subsstem.

:ou s*ould also monitor reads,second5 and )rites,second.

2*is is measured in bloc+s. i.e number o$ bloc+s read,)rite
per second. 2*ese are also re$erred as bi and bo 0bloc+ in
and bloc+ out1.

tps indicates total transactions per seconds5 )*ic* is sum

o$ rtps 0read transactions per second1 and )tps 0)rite
transactions per seconds1.

Disk I/O optimization

Iilesstem to useB

;'2>

;'23

EeiserIS

;'2@

"2EIS
%se IS options in $stab 0noatime5 ...1

Disk I/O optimization

E6!D 0so$t)are or *ard)are1

E6!D=

E6!D1

E6!D? 0not recomended1

E6!D1=

Disk optimization

Use benchmark programs like Bonnie++

Use hdparm

Upgrade BIOS of your server and Firmware of your disk

Does your disk park heads?

Network Tuning

6 good understanding o$ 28/,!/ concepts is *elp$ul

)*ile analzing an net)or+ issues.

Ior net)or+ inter$aces5 ou s*ould monitor total

number o$ pac+ets 0and btes1 received,sent t*roug*
t*e inter$ace5 number o$ pac+ets dropped5 etc.5

TCP Tuning
For servers ha are serving up huge numbers of concuren sessions! here are some cp opions ha
should probabaly be enabled" #ih a large $ of cliens doing heir bes o kill he server! is probabaly
no uncommon for he server o have %&&&& or more open sockes"
'llows more local pors o be available
echo 1024 65000 > /proc/sys/net/ipv4/ip_local_port_range
Increasing he amoun of memory associaed wih socke buffers can ofen improve performance
echo 262143 > /proc/sys/net/core/rmem_max
echo 262143 > /proc/sys/net/core/rmem_default
(hese reduce he amoun of work he ()* sack has o do! so is ofen helpful in his siuaion
echo 0 > /proc/sys/net/ipv4/tcp_sac
echo 0 > /proc/sys/net/ipv4/tcp_timestamps

TCP Tuning (10GB NIC)
# increase TCP max buffer size settable using setsockopt(
net.core.rmemKma4 L 16AAA>16
net.core.)memKma4 L 16AAA>16
# increase !inux autotuning TCP buffer limit
net.ipv@.tcpKrmem L @=96 <A3<= 16AAA>16
net.ipv@.tcpK)mem L @=96 6??36 16AAA>16
# increase the length of the processor input queue
net.core.netdevKma4Kbac+log L 3====
# recommended default congestion control is htcp
net.ipv@.tcpKcongestionKcontrolL*tcp
# recommended for hosts "ith #umbo frames enabled
net.ipv@.tcpKmtuKprobingL1

TCP Congestion Avoidance
Algorithms

renoB 2raditional 28/ used b almost all ot*er operating

sstems. 0de$ault1

cubicB 8%"!8(28/

bicB "!8(28/

*tcpB Mamilton 28/

vegasB 28/ Negas

)est)oodB optimized $or loss net)or+s

Which do u use?
davorg@+ranc*BOP sudo ssctl net.ipv@.tcpKavailableKcongestionKcontrol
net.ipv@.tcpKavailableKcongestionKcontrol L cubic reno

Memory Optimization

!$ ou *ave 16 G" E6# installed on our sstem5 ou *ave 16

G" o$ p*sical memor

Nirtual memor L S)ap space available on t*e dis+ Q /*sical

memor. 2*e virtual memor contains bot* user space and
+ernel space

%sing eit*er 3>(bit or 6@(bit sstem ma+es a big di$$erence in

*o) muc* memor a process can utilize

3n a 3>(bit sstem a process can onl access a ma4imum o$

@G" virtual memor

3n a 6@(bit sstem t*ere is no suc* limitation

More about Memory optimization ...

%nused E6# )ill be used as $ile sstem cac*e b t*e +ernel

Linu4 sstem )ill s)ap )*en it needs more memor. i.e )*en it needs
more memor t*an t*e p*sical memor

-*en it s)aps5 it )rites t*e least used memor pages $rom t*e p*sical
memor to t*e s)ap space on t*e dis+

Lot o$ s)apping can cause per$ormance issues5 as t*e dis+ is muc*

slo)er t*an t*e p*sical memor5 and it ta+es time to s)ap t*e memor
pages $rom E6# to dis+

Memory optimization

Dense #emor 0Mard)are speci$ic1

&%#6 0&on %ni$orm #emor 6ccess1

Muge /ages

#anage Nirtual #emor pages

Ilus*ing o$ dirt pages

S)apping be*avior

NUMAD

%ser(level daemon to automaticall improve out o$ t*e bo4 &%#6 sstem

per$ormance

Iedora 1A

EM;L 6.3 as tec* previe)

&ot enabled b de$ault

#onitors available sstem resources on a per(node basis and assigns signi$icant

consumer processes to aligned resources $or optimum &%#6 per$ormance.

Eebalances )*en necessar

/rovides pre(placement advice $or t*e best initial process placement and
resource a$$init.

Huge Pages

># pages vs @D standard linu4 page

Nirtual to p*sical page map is ?1> times smaller

2L" can map more p*sical pages5 resulting in $e)er misses

2raditional Muge /ages al)as pinned

2ransparent Muge /ages in EM;L6

#ost databases support Muge pages

1G pages supported on ne)er *ard)are

Mo) to con$igure Muge /ages 016G1
ec*o <19> R ,proc,ss,vm,nrK*ugepages
vi ,etc,ssctl.con$ 0vm.nrK*ugepagesL<19>1

Flushing Caches

Drop unused 8ac*e

Irees unused memor

Iile cac*e

!$ t*e D" uses cac*e5 ma notice slo)do)n

Iree pagecac*e
ec*o 1 R ,proc,ss,vm,dropKcac*es

Iree slabcac*e
ec*o > R ,proc,ss,vm,dropKcac*es

Iree pagecac*e and slabcac*e

ec*o 3 R ,proc,ss,vm,dropKcac*es

Swappiness

8ontrols *o) aggressivel t*e sstem reclaims

GmappedS memorB

De$ault ( 6=F

DecreasingB more aggressive reclaiming o$

unmapped pagecac*e memor

!ncreasingB more aggressive s)apping o$ mapped

memor

80/20

$emember the %&/'& rule

<=F o$ t*e per$ormance improvement comes $rom
tuning t*e application5 and t*e rest >=F comes $rom
tuning t*e in$rastructure components.

System Monitoring Tools

vmstat

netstat

ps

top

atop5 *top

mtop

!ostat

4osvie)

Kernel Tuning

+ecompile your kernel

,-clude unneeded modules

Use +eal(ime kernel

.ake kernel smaller

Samba tuning

enable 6!3 0;'2E;#;1

soc+et options L 28/K&3D;L6:

!/23SKL3-D;L6: S3KE8N"%IL6??36
S3KS&D"%IL6??36

ra) read and )rite

oportunistic lo+cing

log level

Database optimization

#SJL 9 use #SJL tuning 6pp

3ptimize table

Do ou use !nnoD" or #!SS6# D".

8*ange database i$ possible

OpenLDAP Tuning

2*e most important tuning aspect $or 3penLD6/ is deciding )*at attributes
ou )ant to build inde4es on.
8ac*esize 1====
dbcac*esize 1=====
sizelimit 1====
loglevel =
dbcac*e&o-snc
inde4 cn5uid
inde4 uidnumber
inde4 gid
inde4 gidnumber
inde4 mail

!$ ou add t*e $ollo)ing parameters to ,etc,openldap,slapd.con$ be$ore entering t*e in$o

into t*e database5 t*e )ill all get inde4ed and per$ormance )ill increase.

Apache Tuning
!ae sure you starting a ton of initial daemons if you "ant good #enchmar scores$
%omething lie&
.inSpareServers %&
.a-SpareServers /&
SarServers 0%
this can #e higher if apache is recompiled
.a-)liens %12
.a-+e3uess*er)hild 4&&&&

'ote& Saring a massive amoun of hpd processes is really a benchmark hack" In mos real world cases! seing
a high number for ma- servers! and a sane spare server seing will be more han ade3uae"
I5s 6us he insan on load ha benchmarks ypically generae ha he SarServers helps wih"

Slow websites

%se optimizers

%se memcac*e

2une our apac*e

#inimize number o$ 6pac*e modules

8*ange 6pac*e $or &gin4

Benchmark

6 good set o$ benc*mar+ing utilities are o$ten ver

*elp$ul in doing sstem tuning )or+. !t is impossible
to duplicate Treal )orldT situations5 but t*at isnt reall
t*e goal o$ a good benc*mar+.

6 good benc*mar+ tpicall tries to measure t*e

per$ormance o$ one particular t*ing ver accuratel.

!$ ou understand )*at t*e benc*mar+s are doing5

t*e can be ver use$ul tools.

Benchmark Tools

bonnie(( ( is a $ree $ile sstem benc*mar+ing tool $or %ni4(li+e operating

sstems

)*ench ( is a tool to generate !,3 )or+loads to eit*er a $ilesstem or to a

net)or+ed 8!IS or &IS server

http+load ( runs multiple *ttp $etc*es in parallel5 to test t*e t*roug*put o$ a

)eb server

dkftpbench ( measuring *o) man simultaneous dialup users can be do)n

loading $rom an I2/ site at t*e same time

tiobench ( is a multi(t*readed !,3 benc*mar+

ttcp ( is a utilit program $or measuring net)or+ t*roug*put

netperf 9 net)or+ per$ormance tester

Identify and solve performace issue

%nderstand t*e problem

Mal$ o$ t*e problem is solved )*en ou clearl understand )*at

t*e problem is.

#onitor and collect data

6$ter de$ining t*e problem clearl5 monitor t*e sstem and tr to

collect as muc* data as possible on various subsstems

;liminate and narro) do)n issues

6$ter *aving a list o$ potential issues5 dive into eac* one o$ t*em
and eliminate an non issues

#a+e one c*ange at a time

donUt tr to ma+e multiple c*anges at one time

TNX ...

;(mailB dguttierrez@me.com

8ompanB ))).3Gen.si

"logB ))).d(mas*ina.net

8NB ))).guttierrez.org