Nginx Performance Tuning: How to do it

9
Submitted by Sarath Pillai on Tue, 12/24/2013 - 16:31

Nginx is a well known web server, and is adopted by many major players in the industry.
The main reason for its fast adoption was that its so fast compared to other web servers
(like Apache). Basically nginx was made to solve a problem that is known as c10k. It
performs much better than any other web server's in the market out of the box for many.
In this article we will see how you can modify your nginx configuration to give it a boost
or say performance tune nginx.
We will get inside the configuration part a little later, coz there are quite a few concepts
that needs to be understood first.
C10k refers to the method of optimizing network connections so that it can handle
connections in the range of ten thousand simultaneously. There is a very famous article
explaining this in the below link.
Read: What is c10k Problem
Apache works in a blocking I/O model. In layman terms it means that when a read/write
request is issued it blocks other requests till that request is completed. A best solution to
this problem is to create separate threads/processes for each connections. This is what
Apache does.
Apache works by using a dedicated thread per client with blocking I/O.

Although dedicated thread per connections/request by Apache is a good method to serve

clients, its a memory consuming as well as processor consuming method.
Its processor consuming, because for each request that hits the Apache server, the
server processor has to switch between different processes (because each http requests
connections creates a new process/thread)

Related: Process administration in Linux

Now if performance and resource utilization is not at all a concern for you, then you can
go ahead with the thread per connection model of Apache by using an Apache web
server to serve requests. But if you are concerned about both resource utilization and
performance, then nginx model of event driven architecture is the best to go ahead with.

Nginx uses a single threaded non-blocking I/O mechnism to serve requests. As it uses
non-blocking I/O, one single process can server too many connection requests.

Related: Nginx and Apache difference

As everything is a file in linux, even network connections are files internaly. Each and
every process has its own set of file descriptors that needs to be polled frequently to
identify which is one is ready to be read from or write to (Even connections that an nginx
server receives are maintained by the nginx process in the form of file descriptors for the
life time of a connection).
As we discussed before, nginx uses a non-blocking I/O with a single thread. Hence the
single process must identify which connection file is ready to be read or written to. For
this, the operating system has three different methods. They are mentioned below.

Select Method
Poll Method
Epoll Method
Using both select and poll is not an efficient way to identify which file descriptor is ready.
Because both of these method are identical in what they does (am not capable enough to
explain you the difference between select and epoll, if you are a programmer i will
recommend to go through the below link to learn more about select and poll.)
Read: Select and Poll
So nginx can use either select or poll method to identify which file(file descriptor) is
ready to be read or written to. But both these methods are not efficient enough when
there are too many number of connections. For example, if you want to serve say 10000
connections at a time. And only one of the them is ready to be read. To identify that one
file descriptor to which is ready to be read, the process has to still scan through rest of
the 9999 file descriptors (which is waste of resource).

Another method other than the select and Poll is called as epoll comes to rescue here.
This is only available in Linux kernels which are later than 2.6. Epoll uses an efficient
mechanism compared to poll and select.
If you have 10000 connections made to your server, and 8000 among them are idle
connections. Using poll and select is not that efficient because epoll will only give
importance to connections that are active. Please note the fact that all three select, poll
and epoll basically does the same thing, but using epoll is less cpu intensive when you
have to serve thousands of connections.

You can modify your nginx configuration file to use epoll as shown below. If you are on
centos this nginx configuration file will be located at /etc/nginx/nginx.conf. This is done
inside an events block in the configuration file.
?

1
2
3

events {
use epoll;
}

Now as discussed earlier, nginx works on a non-blocking single process model. Although
we say single process, we can ask our nginx to start multiple worker process. Basically its
like if you have two worker process, each one will handle all the requests. This setting is
normally configured depending upon the number of CPU cores you have on your system.
You can find the number of CPU cores on your system by using either of the below
commands.
?
1
2
3
4
5

[root@www ~]# lscpu

Architecture:
x86_64
CPU op-mode(s):
32-bit, 64-bit
Byte Order:
Little Endian
CPU(s):
8

?
1

cat /proc/cpuinfo

Am quite sure that if you are on a production server, you will be having multiple cpu
cores. As seen from the lscpu command output, i have 8 processors, so i will ask nginx to
launch 8 worker process that will be serving requests.
?
1

worker_processes 8;

Also you can alternatively configure nginx to do the job of identifying the number of cores
on your server and launch worker processes accordingly. This can be done by modifying
the worker_processes to auto as shown below.

?
1

worker_processes auto;

Another important factor that we can modify in nginx is the maximum number of files
that can be opened by our worker processes. As discussed before, each and every
connection is handled by creating files to identify them. Hence more the number of files
that are allowed to be be created, workers can work with more connections.
There is a configuration value that can be modified in nginx to get this done. Its always
good to set this value to a higher one if your server is a very high traffic one.
?
1

worker_rlimit_nofile 100000;

Even if i set worker_process in my nginx.conf to 8 or auto, it will end up launching 8

worker processes. All these 8 processes are ready to serve connections. But you can
further optimize these worker process to make them server as many connections as
possible.
We can define the maximum number of connections which can be simultaniously served
by a single worker process. The default is 512 connections if you dont specify it. But you
can modify this to accept more connections per worker. This can be done as shown
below.
?
1
2
3
4

events {
use epoll;
worker_connections 1024;
}

We also need to configure our worker processes to accept multiple connections at one
time. This can be done by modifying the events block as shown below.

?
1
2
3
4
5

events {
use epoll;
worker_connections 2048;
multi_accept on
}

Our final configuration after making the above changes must look something like the
below.
?
1
2
3
4
5
6
7
8

worker_processes auto;
worker_rlimit_nofile 100000;
events {
worker_connections 1024;
use epoll;
multi_accept on;
}

So now my nginx web server is ready to serve around 8192 connections(8 worker
process * 1024 connections ) at one time.

Please note the fact that we have used events module in nginx to modify worker
connections, use epoll method, and multi_accept. This is the reason why they are inside
events block (events {}) in the configuration file. There is another module called http,
which is the primary module that you will be using to define http related stuff.

We wil be now modifying and adding few parameters inside the http module to speed up
our nginx a little bit more.
One of the major and required feature of any webserver is logging. Logging will give the
server administrator an indepth detail of the kind of requests that are being served, and
also it helps for troubleshooting. A web server log can also be used to make a web
analytics because log files contain the complete details of the requests. These details
include the source address of the request, user agent (the browser used by the client),
the resource requested in the request (or the url accessed), destination address, referral
url etc etc.
Related: Central Logging Server in Linux
Although logging is an important and required feature, it does a lot of I/O operation on
the disk. Please dont forget the fact that each and every request is logged as soon as it
hits the server and served.
This means for every request the web server has to process the logs and add the
relevant details to the log file (which adds up to processing as well as I/O usage.). If the
website that you host is getting too much traffic, then am sure your web server is having
a realy bad time logging it.
Hence turning off the logs completely on the web server can save a lot of I/O as well as
CPU power. This becomes very handy when you are having a high traffic website. In
nginx you can completely turn of logging by the following option.
?
1
2
3
4
5

http {
include
/etc/nginx/mime.types;
default_type application/octet-stream;
access_log off;
}

access_log off option shown in the above http block tells not to log requests that are
being processed by the server.

When you access a website with a web browser, what really happens is that your browser
establishes an HTTP connection with the target server. The real overhead is the
beginning. Which means the main time consuming (although in seconds, it matters when
performance is considered) thing is to establish the connection.
Let's take an example, where you are accessing a news related web site, and you are
flipping through different url's on the website to have a quick overview of different news
contents. Imagine a situation where your client browser has to make new http
connections to the server for each URL you are accessing on the site. That will be a little
slow for the site visitors. Because accessing each url has another overhead of
establishing an HTTP connection first.

HTTP version 1.0 implemented a feature called as keep alive connections to solve this
overhead of creating a connection for each request. However there is a timeout period
for this. In short the web server's allow you to specify the timeout period for which to
keep the connections alive, so that a connection made during that interval does not
require to create a connection.
In short connections are kept alive till the keep alive timeout you specify

And believe me nginx is much better in handling large number of active connections,
compared to any other web servers. It claims that it can handle upto 10000 connections
alive by only utilizing 2.5 MB of memory. So if you are using nginx, go ahead and raise
your keep alive value to something higher like 65.
?
1

keepalive_timeout 65;

If you are a Linux user, then am sure you must have heard about gzip compression. And
you might already know the amount of compression level that can be achieved with gzip
for compressing text content. You get compression rate in the range of 90's. Which is a
very awesome compression level.
A major portion of a web page is always text content. The components that make a web
page look good as well as enhance its functionalities are also text content. Although they
are all code in different languages like html, CSS and even JavaScript (which the browser
knows to interpret) at the end of the day they are text content.
Sending all data as it is over the wire to clients is always a bit slow. So each and every
web server supports compression mechanism to compress the data that are being sent
out to the clients. Also all modern web browsers knows how to decompress it.
Enabling compression can be very helpful to people who are on slow networks accessing
your website. Enabling compression is not a tough task. Similar to the previously shown
options, compression and gzip option is enabled inside the http block section in
nginx.conf file.
If you have done compression with gzip tool in linux, then you must be knowing that it
takes a little bit of time to compress the content depending upon the size. And you might
also know that if you increase the compression level, the end result does not make any
large substancial difference, but it will take a little more time along with a little more CPU
usage.
Similarly while configuring compression on our nginx web server, we need to take an
extra care of what to compress, when to compress, and the level of compression. Also we
need to have a mechanism where we can disable compression for older web browsers
that does not support this feature.
So the first step is to enable compression. This can be done by adding the below
parameter inside the http block.
?

gzip on;

The second step is to tell nginx, to only compress, when the size of the data being sent is
above our specified limit. This can be done as shown below.

?
1

gzip_min_length 256;

Official nginx web site describes the above min length parameter in a nice way. Below is
the description of the min length parameter from the official nginx http module section.

Responses shorter than this byte-length will not be compressed. Length is determined
from the "Content-Length" header.

Now as we discussed earlier, too much compression does not make a substantial
difference, but will only add up to CPU usage. Compression level is mentioned from a
scale of 1 to 9. We do need compression, but not at a higher level. Because even the
basic gzip compression does a pretty nice job. So lets keep the value of compression
level somewhere between from 3 to 5 (Depending upon your wish)

?
1

gzip_comp_level 3;

Now we need to tell our nginx web server about the types of data that needs to be
compressed. This is generally text content ranging from rss, css, html, javascript etc. This
can be done by adding the following line.

?
1

gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml applic

So our final http block section will look something like the one shown below.

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

http {
include
/etc/nginx/mime.types;
default_type application/octet-stream;
access_log off;
keepalive_timeout 65;
gzip on;
gzip_min_length 256;
gzip_comp_level 3;

gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml appl

}

Another aspect of nginx that can substantially improve performance is to enable caching
for metadata. Please note the fact that this does not enable caching for the content, but
its caching for metadata. This feature of nginx (which is again part of the http block
section), is called as Open_file_cache. The first thing to do is to enable it.

?
1

open_file_cache max=10000 inactive=30s;

Please note the fact that there is no ON switch for this, but there is an off switch. Before
we understand the above configuration parameter, let's see what are the contents that
will be cached with this.

Open File Descriptors

File Descriptor modification time, their size etc.
Error status like no permission, no file etc

The first parameter, max=10000 tells nginx web server to only cache this many number
of entries. Old inactive entries are automatically flushed out. Please note the fact that
the inactive switch tells the time after which a cache entry will be removed if inactive.
Other open_file_cache options that needs to be configured are mentioned below.

1.
2.

Validity of a cache entry (open_file_cache_valid).

Minimum number of times the cache entry has to be accessed before
the inactive number of seconds, so that it stays in cache
(open_file_cache_min_uses)

3.

Cache errors while searching a file (open_file_cache_errors)

So our open file cache configuration will look something like the below.

?
1
2
3
4

open_file_cache max=10000 inactive=30s;

open_file_cache_valid 60s;
open_file_cache_min_uses 2;
open_file_cache_errors on;

If you have a very high traffic web site with nginx, setting the above open file cache
parameters properly can really boost the performance.

Although we did see the keep alive stuff in nginx, there are yet few more parameters that
is very helpful in specifing the number of connections to be kept alive from a client. Also
we have few more options to close connections when the client does not respond.

The first one is to limit the total number of requests that can be served through a keep
alive connection. The default value if not mentioned is 100 (which means a client can
make 100 successfull request inside one keep alive connection.). Even that is too high for
one client, if you want you can increase this value to something like 200 or something.
This can be done as shown below.

?
1

keepalive_requests 200;

Keeping connections open for a client which is not responding is not a good idea, its
always better to close that connection and free up memory associated with it. This can
be done as shown below.

?
1

reset_timedout_connection on;

Another nice option that needs to enabled for faster tcp data transfer is sendfile. There is
an excellent article from techrepublic, that explains the entire stuff in detail.

Read: What is sendfile() and how is it useful

This option can be enabled in nginx as shown below.

?
1
2

sendfile on;
tcp_nopush on;

tcp_nopush option will make nginx to send all header files in a single packet rather than
seperate packets.

So our final nginx.conf file will look something like the below shown one.

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

worker_processes auto;
worker_rlimit_nofile 100000;
events {
use epoll;
worker_connections 1024;
multi_accept on
}
http {
include
/etc/nginx/mime.types;
default_type application/octet-stream;
access_log off;
keepalive_timeout 65;
keepalive_requests 200;
reset_timedout_connection on;
sendfile on;
tcp_nopush on;
gzip on;
gzip_min_length 256;
gzip_comp_level 3;
gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml ap
open_file_cache max=10000 inactive=30s;
open_file_cache_valid 60s;
open_file_cache_min_uses 2;
open_file_cache_errors on;
}

Once you have configured the required changes to improve the speed, i would suggest to
do a web server performance test, with AB tool or any other tool to confirm its serving
the purpose.

Read: How to do web server performance test

Please let me know if this article was helpful. There are more number of options in nginx,
i will surely update this article from time to time, as i find or learn something new about
this. Also please do not forget to point out any wrong or incorrect information if any is
present in this article through comments (as this will help us as well as our readers)