Blogzor

Zach Orr's Blogosphere Thingy.

Battle Ready Nginx - an Optimization Guide

Most setup guides for Nginx tell you the basics – apt-get a package, modify a few lines here and there, and you’ve got a web server! And, in most cases, a vanilla nginx install will work just fine for serving your website. However, if you’re REALLY trying to squeeze performance out of nginx, you’ll have to go a few steps further. In this guide, I’ll explain which settings in nginx can be fine tuned in order to optimize performance for handling a large number of clients. As a note, this isn’t a comprehensive guide for fine-tuning. It’s a brief overview of some settings that can be tuned in order to improve performance. Your mileage may vary.

Basic (Optimized) Configuration

The only file we’ll be modifying is your nginx.conf, which holds all your settings for nginx in different modules. You should be able to find nginx.conf in the /etc/nginx directory on your server. First, we’ll talk about some of the global settings, then go through each module in the file and talk about which settings will get you the best performance for a large number of clients, and why they’ll increase your performance. A completed config file can be found at the end of this post.

Top Level Configs

Nginx has a few top level configuration settings that sit outside the modules in your nginx.conf file.

1
2
3
4
5
6
user www-data;
pid /var/run/nginx.pid;

worker_processes auto;

worker_rlimit_nofile 65000;

user and pid should be set by default – we won’t modify this, since it won’t do anything for us

worker_processes defines the number of worker processes that nginx should use when serving your website. The optimal value depends on many factors including (but not limited to) the number of CPU cores, the number of hard drives that store data, and load pattern. When in doubt, setting it to the number of available CPU cores would be a good start (the value “auto” will try to autodetect it).

worker_rlimit_nofile changes the limit on the maximum number of open files for worker processes. If this isn’t set, your OS will limit. Chances are your OS and nginx can handle more than ulimit -n will report, so we’ll set this high so nginx will never have an issue with “too many open files”. Make sure to update these values on your OS, also.

Events Module

The events module contains all the settings for processing connections in nginx.

1
2
3
4
5
6
7
events {
    worker_connections 2048;

    multi_accept on;

    use epoll;
}

worker_connections sets the maximum number of simultaneous connections that can be opened by a worker process. Since we bumped up worker_rlimit_nofile, we can safely set this pretty high.

multi_accept tells nginx to accept as many connections as possible after getting a notification about a new connection

use sets which polling method we should use for multiplexing clients on to threads. If you’re using Linux 2.6+, you should use epoll. If you’re using *BSD, you should use kqueue. Alternatively, if you don’t include this parameter nginx will choose for you. Wanna know more about event polling? Let Wikipedia be your guide (warning, a neckbeard and an operating systems course might be needed to understand everything).

HTTP Module

The HTTP module controls all the core features of nginx’s http processing. Since there are quite a few settings in here, we’ll take this one in pieces. All these settings should be in the http module, even though it won’t be specifically noted as such in the snippets.

1
2
3
4
5
6
7
8
9
http {

    sendfile on;

    tcp_nopush on;
    tcp_nodelay on;

    ...
}

sendfile enables the use of sendfile(). sendfile() copies data between the disk and a TCP socket (or any two file descriptors). Pre-sendfile, to transfer such data we would alloc a data buffer in the user space. We would then read() to copy the data from a file in to the buffer, and write() the content of the buffer to a network. sendfile() reads the data immediately from the disk into the OS cache. Because this copying is done within the kernel, sendfile() is more efficient than the combination of read() and write() and the context switching/cache trashing that comes along with it (read more about sendfile).

tcp_nopush tells nginx to send all header files in one packet as opposed to one by one.

tcp_nodelay tells nginx not to buffer data and send data in small, short bursts – it should only be set for applications that send frequent small bursts of information without getting an immediate response, where timely delivery of data is required.

1
2
3
access_log off;

error_log /var/log/nginx/error.log crit;

access_log sets whether or not nginx will store access logs. Turning this off increases speed by reducing disk IO (aka, YOLO).

error_log tells nginx it should only log critical errors.

1
2
3
4
5
6
7
8
keepalive_timeout 20;

client_header_timeout 20;
client_body_timeout 20;

reset_timedout_connection on;

send_timeout 20;

keepalive_timeout assigns the timeout for keep-alive connections with the client. The server will close connections after this time. We’ll set it low, since we can only open up so many file descriptors (connections) at once.

client_header_timeout and client_body_timeout sets the timeout for the request header and request body (respectively). We’ll set this low too.

reset_timedout_connection tells nginx to close connections on non-responding clients. This will free up all memory associated with that client.

send_timeout specifies the response timeout to the client. This timeout does not apply to the entire transfer, but between two subsequent client-read operations. If the client has not read any data for this amount of time, then nginx shuts down the connection.

1
2
3
4
5
include /etc/nginx/mime.types;

default_type text/html;

charset UTF-8;

include is just a directive to include the contents of another file in the current file. Here, we use it to load in a list of MIME types to be used later.

default_type sets the default MIME-type to be used for files.

charset sets the default charset to be included in our header.

The performance improvement these two options give is explained in this great WebMasters StackExchange question.

1
2
3
4
5
6
7
8
9
10
11
gzip on;

# gzip_static on;

gzip_proxied any;

gzip_min_length 256;

gzip_comp_level 4;

gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

gzip tells nginx to gzip the data we’re sending. This will reduce the amount of data we need to send.

gzip_static tells nginx to look for the pre-gzip’d asset with the same name before gzipping the asset itself. This is requires you to pre-zip your files (it’s commented out for this example), but allows you to use the highest compression possible and nginx no longer has to zip those files (read more about gzip_static here).

gzip_proxied allows or disallows compression of a response based on the request/response. We’ll set it to any, so we gzip all requests.

gzip_min_length sets the minimum number of bytes necessary for us to gzip data. If a request is under 1000 bytes, we won’t bother gzipping it, since gzipping does slow down the overall process of handling a request.

gzip_comp_level sets the compression level on our data. These levels can be anywhere from 1 to 9, 9 being the slowest but most compressed. We’ll set it to 4, which is a good middle ground.

gzip_types sets the type of data to gzip. There are some above, but you can add more.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# cache informations about file descriptors, frequently accessed files
# can boost performance, but you need to test those values
open_file_cache max=65000 inactive=20s;
open_file_cache_valid 30s;
open_file_cache_min_uses 2;
open_file_cache_errors on;

##
# Virtual Host Configs
# aka our settings for specific servers
##

include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}

open_file_cache both turns on cache activity and specifies the maximum number of entries in the cache, along with how long to cache them. We’ll set our maximum to a relatively high number, and we’ll get rid of them from the cache if they’re inactive for 20 seconds.

open_file_cache_valid specifies interval for when to check the validity of the information about the item in open_file_cache.

open_file_cache_min_uses defines the minimum use number of a file within the time specified in the directive parameter inactive in open_file_cache.

open_file_cache_errors specifies whether or not to cache errors when searching for a file.

include is again used to add some files to our config. We’re including our server modules, defined in a different file. If your server modules aren’t at these locations, you should modify this line to point at the correct location.

The full config file

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
user www-data;
pid /var/run/nginx.pid;
worker_processes auto;
worker_rlimit_nofile 65000;

events {
    worker_connections 2048;
    multi_accept on;
    use epoll;
}

http {
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;

    access_log off;
    error_log /var/log/nginx/error.log crit;

    keepalive_timeout 20;
    client_header_timeout 20;
    client_body_timeout 20;
    reset_timedout_connection on;
    send_timeout 20;

    include /etc/nginx/mime.types;
    default_type text/html;
    charset UTF-8;

    gzip on;
    # gzip_static on;
    gzip_proxied any;
    gzip_min_length 256;
    gzip_comp_level 4;
    gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

    open_file_cache max=65000 inactive=20s;
    open_file_cache_valid 30s;
    open_file_cache_min_uses 2;
    open_file_cache_errors on;

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
}

After editing your config, make sure to reload/restart nginx in order to have it use our new configuration file. With upstart, run

sudo service nginx reload

Takeaway

There we go! Your web server is now ready to do battle with the army of visitors that previously plagued you. This is by no means the only way you can go about speeding up your website. I’ll be writing more posts that explain other ways to speed up your website soon.

Have feedback? Shoot me an email or discuss this post on HackerNews.