Apache Optimization and NGINX

Once again, my WordPress friends had to hit me up for some help. These are the same friends that I helped with a Custom Field Search Plugin. They have been maintaining a community site, SuccessNet Online™, and an email mailing list of several hundred thousand participants for Business Networking International.

They send out an email every month with a quick update on the community and directing participants to visit the site. For several months they had been using a Lyris server in their office to send out the mailing list. This past month, they started using Lyris HQ to send out the newsletter. When the first newsletter was sent out using the Lyris HQ, the community site quickly ground to a halt and nearly crashed the server. They quickly gave me a call to figure out why the site was having problems and what we could do to prevent it from happening again.

The reason for the site performance issues was immediately apparent once they told me that they were no longer using the Lyris server in their office. They went from an 3-year old server sitting behind a T1 to a managed service sitting behind who knows how big of a pipe. Instead of taking 7-8 hours to deliver the email newsletter using the office server, the new service was able to do it in about 15 minutes! As a result, the peak traffic to the site was compressed from 12 hours to a single hour. Using the default Apache configuration, there was no way they could handle the traffic. With about an hour of work, I was able to optimize their server configuration to handle the traffic. Because I was able to do this without increasing the resource footprint of their server, my friends were very happy.

This same story is repeated time-and-again by businesses large and small that encounter sudden spikes of traffic. The reason for the spike can range from a Slashdotting to the removal of a resource bottleneck. The result is nearly always the same and exposes a fundamental weakness in common Apache configurations.

Anatomy of an Apache Failure
The default configuration of Apache is set up to handle the delivery of static files. This default configuration is able to deliver massive amounts of content within a small footprint. When you begin adding an application layer and a database to the mix, you introduce the ability for Apache to bring even high-end servers to their knees.

Most web applications that run under Apache require the use of the PMP prefork configuration. These include web applications written in PHP, mod_perl, mod_python, etc. The critical section of the Apache configuration that we are going to look follows:

<IfModule mpm_prefork_module>
    StartServers          5
    MinSpareServers       5
    MaxSpareServers      10
    MaxClients          150
    MaxRequestsPerChild   0
</IfModule>

When this configuration is placed under load, Apache will create up to 150 worker processes to handle the requests. A typical worker process that handles a WordPress site will weigh in at about 30MB of memory. In order for a web server to handle 150 of these workers, it would need at least 4GB of memory! This is also a very optimistic estimate. I’ve seen quite a few site applications that end up with workers using 200MB of memory and up. The typical web server has no where near enough memory to handle 150 workers.

At idle, this Apache configuration will have ten idle worker processes. Now we start adding traffic, measured in concurrent requests. Once there are 6 concurrent requests, Apache creates an additional worker because it is required to have a minimum of 5 spare workers. The creation of a new worker process is not free and it does have an impact on the load of the server. As load continues to increase, Apache will continue to create worker processes. Once all physical memory is exhausted, additional workers will begin to hit swap memory. The overhead required to use the swap memory imparts even more load on the system. Once this point is reached, worker creation will snowball out of control. One of two things will typically happen—and happen fast: the server enters swap-death; or the database crashes under too many connections.

We have all seen both on many occasions. In the case of swap-death, the browser connection times out. In the case of a crashed database, we get the standard ‘unable to connect to database’ error. So how do we avoid that happening on our own site?

Strict Apache Resource Limits
The only sure-fire method to preventing load from disabling the web server is to configure Apache so that it will not consume more resources than are available. In the case of the WordPress site I was called in to help on, the server had 1GB of memory and was running both Apache and MySQL. My plan called for having Apache consume no more than 2/3 of the physical memory of the system, leaving the remaining 1/3 for MySQL and the OS. Using the previous 30MB estimate, this allows for a maximum of 20 worker processes:

<IfModule mpm_prefork_module>
    StartServers          10
    MinSpareServers       10
    MaxSpareServers       10
    MaxClients            20
    MaxRequestsPerChild    0
</IfModule>

As you can see, I also increased the start, min, and max number of processes. I did this to further reduce the frequency of worker creation and destruction—reducing overhead as load increases and decreases. For high traffic sites that utilize a dedicated database server, I recommend setting the start, min, and max servers to max clients.

With the changes made, it is time to test things out. My personal preference is to fire up several looping wget scripts that also download the images, javascript, and stylesheets for the page:

#!/bin/bash
while true; do
wget -pq http://www.server.com/
done

With the resource limits in place, the system load will remain at acceptable levels even with several scripts generating traffic. However, as the number of traffic scripts are increased, the responsiveness of Apache decreases at an alarming rate. When I used this method on the WordPress site, I was experiencing delays with as few as two scripts generating traffic.

Introducing a Reverse Proxy
Now we have Apache in a configuration that no longer runs away with our resources, but the site appears to be significantly slower to respond to requests. The solution is to have separate web server instance to serve all static content. The easiest way to accomplish this is through the use of a reverse proxy. The reverse proxy responds to all web requests to the site. If the request is for static content, it delivers the content. If the request is for dynamic content, it requests the dynamic content from Apache. This ensures that the heavyweight Apache worker processes are only being used to respond to dynamic content requests and a lightweight process is being used to respond to static requests. Otherwise, the heavyweight workers will spend most of their time responding to static requests.

I cannot stress the importance of a reverse proxy enough for high volume sites! Without a reverse proxy, our small pool of heavyweight processes spend a significant amount of time just sending the raw data for the static content. By having a separate web server send the data for static content, you minimize the impact that data delivery will have in tying up worker processes.

My server of choice for the role of reverse proxy is NGINX. NGINX is easy to configure, fast, and stable. The first step of installation is to reconfigure Apache to listen to a port other than 80. Port 8000 or 8080 are commonly used. Once that change has been made and confirmed, NGINX can be installed.

I prefer to set up NGINX initially to proxy all content from port 80 to port 8000. This allows me to verify that everything is working correctly before I set up the static handlers. I then add a proxy for the specific hosts that I need. An edited version of the configuration that I use on this site follows:

nginx.conf

user                    www-data;
worker_processes        5;
error_log               /var/log/nginx/error.log;
pid                     /var/run/nginx.pid;
worker_rlimit_nofile    8196;

events {
    worker_connections    1024;
}

http {
    include         /usr/local/nginx/conf/mime.types;
    include         /usr/local/nginx/conf/proxy.conf;
    default_type    application/octet-stream;

    access_log  /var/log/nginx/access.log;

    sendfile        on;
    tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  65;
    tcp_nodelay        on;

    gzip  on;

    include /usr/local/nginx/conf/sites/*;
}

proxy.conf

proxy_redirect          off;
proxy_set_header        Host            $host;
proxy_set_header        X-Real-IP       $remote_addr;
proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
client_max_body_size    10m;
client_body_buffer_size 128k;
proxy_connect_timeout   90;
proxy_send_timeout      90;
proxy_read_timeout      90;
proxy_buffers           32 4k;


sites/000-default:
server {
    listen 80;
    server_name _;

    location / {
        proxy_pass  http://127.0.0.1:8000;
    }
}

sites/braindonor.net:

server {
    listen 80;
    server_name braindonor.net *.braindonor.net;

    # serve static files directly
    location ~* ^.+.(jpg|jpeg|gif|css|png|js|ico)$ {
        root /var/www/www.braindonor.net/htdocs/;
    }

    # proxy the rest
    location / {
        proxy_pass  http://127.0.0.1:8000;
    }
}

Be sure to check both the Apache and NGINX log files to ensure that requests are being handled by the appropriate server.

Wrapping Things Up
With the reverse proxy in place and running, there are a few tasks to wrap up.

The LogFormat in apache.conf needs to be updated to reflect the use of a reverse proxy:

LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b" common

If you are using any log analysis programs, you will need to update them to use the logs from NGINX instead of Apache.

Don’t hesitate to leave a comment if you have questions or need some help!