Letsgetdugg

Since Varnish did not work out on Solaris yet again. I have decided to bite the bullet and write a headers normalization patch for Squid 2.7. This patch should produce much better cache hit rates with Squid. Efficiency++

What the patch does:

1. Removes Cache-Control request headers, don’t let clients by-pass cache if it is primed.
2. Normalize Accept-Encoding Headers for a higher cache hit rate.
3. Clear Accept-Encoding Headers for content that should not be compressed such as image,video and audio.

and the patch: squid-headers-normalization.patch

Update: Fixed a minor memory leak, all good now.
Update 2: Added audio exception to strip accept-encoding.

--- src/client_side.c.og	2010-01-20 12:00:56.000000000 -0800
+++ src/client_side.c	2010-01-19 20:35:31.000000000 -0800
@@ -3983,6 +3983,7 @@
 	    errorAppendEntry(http->entry, err);
 	    return -1;
 	}
+
 	/* compile headers */
 	/* we should skip request line! */
 	if ((http->http_ver.major >= 1) && !httpMsgParseRequestHeader(request, &msg)) {
@@ -3992,10 +3993,59 @@
 	    err->url = xstrdup(http->uri);
 	    http->al.http.code = err->http_status;
 	    http->log_type = LOG_TCP_DENIED;
+
 	    http->entry = clientCreateStoreEntry(http, method, null_request_flags);
 	    errorAppendEntry(http->entry, err);
 	    return -1;
 	}
+
+	/*
+	 *  Normalize Request Cache-Control / If-Modified-Since Headers
+	 *  Don't let client by-pass the cache if there is cached content.
+	 */
+	if(httpHeaderHas(&request->header,HDR_CACHE_CONTROL)) {
+		httpHeaderDelByName(&request->header,"cache-control");
+	}
+
+	/*
+	 * Un-comment this if you want Squid to always respond with the request
+	 * instead of returning back with a 304 if the cache has not changed.
+	 */
+	/*
+	if(httpHeaderHas(&request->header,HDR_IF_MODIFIED_SINCE)) {
+		httpHeaderDelByName(&request->header,"if-modified-since");
+	}*/
+
+	/*
+	 * Normalize Accept-Encoding Headers sent from client
+	 */
+	if(httpHeaderHas(&request->header,HDR_ACCEPT_ENCODING)) {
+		String val = httpHeaderGetByName(&request->header,"accept-encoding");
+		if(val.buf) {
+			if(strstr(val.buf,"gzip") != NULL) {
+				httpHeaderDelByName(&request->header,"accept-encoding");
+				httpHeaderPutStr(&request->header,HDR_ACCEPT_ENCODING,"gzip");
+			} else if(strstr(val.buf,"deflate") != NULL) {
+				httpHeaderDelByName(&request->header,"accept-encoding");
+				httpHeaderPutStr(&request->header,HDR_ACCEPT_ENCODING,"deflate");
+			} else {
+				httpHeaderDelByName(&request->header,"accept-encoding");
+			}
+		}
+		stringClean(&val);
+	}
+
+	/*
+	 * Normalize Accept-Encoding Headers for video/image content
+	 */
+	char *mime_type = mimeGetContentType(http->uri);
+	if(mime_type) {
+		if(strstr(mime_type,"image") != NULL || strstr(mime_type,"video") != NULL || strstr(mime_type,"audio") != NULL) {
+			httpHeaderDelByName(&request->header,"accept-encoding");
+		}
+	}
+
+
 	/*
 	 * If we read past the end of this request, move the remaining
 	 * data to the beginning

Clearing stale cache by domain

You can clear a site’s cache by domain, this is really nifty if you have Varnish in front of multiple sites. You can log into Varnish’s administration console via telnet and execute the following purge command to wipe out the undesired cache.

purge req.http.host ~ letsgetdugg.com

Monitor Response codes

Worried that some of your clients might be receiving 503 Varnish response pages? Find out with varnishtop.

varnishtop -i TxStatus

Here is how the output looks like.

list length 7                                                                                                                                                                               web

  4018.65 TxStatus	 200
   132.35 TxStatus	 304
    44.17 TxStatus	 404
    34.63 TxStatus	 302
    30.87 TxStatus	 301
     9.36 TxStatus	 403
     1.39 TxStatus	 503

Update 2010-02-19: Seems other people are also affected by the Varnish LINGER crash on OpenSolaris. This does not address the core problem but removes the “fail fast” behavior with no negative side effects.

r4576 has been running reliably with the fix below.

In varnishd/bin/cache_acceptor.c

Remove TCP_assert line encapsulating setsockopt().

Update 2010-02-17: This might be a random fluke but Varnish has connection issues when compiled under SunCC, stick to GCC. I have compiled Varnish with GCC 4.3.2 and the build seems to work well. Give r4572 a try, phk commited some solaris aware errno code.

Update 2010-02-16: r4567 seems stable. Errno isn’t thread-safe by default on Solaris like other platforms, you need to define -pthreads for GCC and -mt for SunCC in both the compile and linking flags.

GCC example:

SunCC Example:

Here are the sources on how I pieced it all together: sun docs, stack overflow answer

See what -pthreads define on GCC

snippet from solaris’s /usr/include/errno.h to confirm that errno isn’t thread safe by default.

Update 2010-01-28: r4508 seems stable. No patches needed aside from removing an assert(AZ) in cache_acceptor.c on line 163.

Update 2010-01-21: If your using Varnish from trunk past r4445 apply this session cache_waiter_poll patch to avoid stalled connections.

Update 2009-21-12: Still using Varnish in production, the site is working beautifully with the settings below.

Update(new): I think I figured the last remaining piece of the puzzle. Switching Varnish’s default listener to poll fixed the long connection accept wait times.

Update: Monitor charts looked good, but persistent connections kept flaking under production traffic. I was forced to revert back to Squid 2.7. *Sigh* I think Squid might be the only viable option on Solaris when it comes to reverse proxy caching. The information below is useful if you still want to try out Varnish on Solaris.

I have finally wrangled Varnish to work reliably on Solaris without any apparent issues. The recent commit to trunk by phk(creator) fixed the last remaining Solaris issue that I am aware of.

There are three four requirements to get this working reliably on Solaris.

1. Run from trunk – r4508 is a known stable revision that works well. Remove the AZ() assert in cache_acceptor.c on line 163.

2. Set connect_timeout to 0, this is needed to work around a Varnish/Solaris TCP incompatibility that resides in lib/libvarnish/tcp.c#TCP_connect timeout code.

3. Switch the default waiter to poll. EventPorts seems bugged on OpenSolaris builds.

4. If you have issues starting Varnish, start Varnish in the foreground via -F argument.

Here is a Pingdom graph of our monitored service. Can you tell when Varnish was swapped in for Squid? Varnish does a better job of keeping content cached due to header normalization and larger cache size.

There are a few “gotchas” to look out for to get it all running reliably. Here is the configuration that I used in production. I have annotated each setting with a brief description.

newtask -p highfile /opt/extra/sbin/varnishd
-f /opt/extra/etc/varnish/default.vcl
-a 0.0.0.0:82                 # IP/Port to listen on
-p listen_depth=8192          # Connections kernel buffers before rejecting.
-p waiter=poll			  # Listener implementation to use.
-p thread_pool_max=2000       # Max threads per pool
-p thread_pool_min=50         # Min Threads per pool, crank this high
-p thread_pools=4             # Thread Pool per CPU
-p thread_pool_add_delay=2ms  # Thread init delay, not to bomb OS
-p cc_command='cc -Kpic -G -m64 -o %o %s'  # 64-Bit if needed
-s file,/sessions/varnish_cache.bin,512M   # Define cache size
-p sess_timeout=10s           # Keep-Alive timeout
-p max_restarts=12            # Amount of restart attempts
-p session_linger=120ms       # Milliseconds to keep thread around
-p connect_timeout=0s         # Important bug work around for Solaris
-p lru_interval=20s           # LRU interval checks
-p sess_workspace=65536       # Space for headers
-T 0.0.0.0:8086               # Admin console
-u webservd                   # User to run varnish as

System configuration Optimizations

Solaris lacks SO_{SND|RCV}TIMEO BSD socket flags. These flags are used to define TCP timeout values per socket. Every other OS has it Mac OS X, Linux, FreeBSD, AIX but not Solaris. Meaning Varnish is unable to make use of custom defined timeout values on Solaris. You can do the next best thing with Solaris; optimize the TCP timeouts globally.

# Turn off Nagle. Nagle Adds latency.
/usr/sbin/ndd -set /dev/tcp tcp_naglim_def 1

# 30 second TIME_WAIT timeout. (4 minutes default)
/usr/sbin/ndd -set /dev/tcp tcp_time_wait_interval 30000

# 15 min keep-alive (2 hour default)
/usr/sbin/ndd -set /dev/tcp tcp_keepalive_interval 900000

# 120 sec connect time out , 3 min default
ndd -set /dev/tcp tcp_ip_abort_cinterval 120000

# Send ACKs right away - less latency on bursty connections.
ndd -set /dev/tcp tcp_deferred_acks_max 0

# RFC says 1 segment, BSD/Win stack requires 2 segments.
/usr/sbin/ndd -set /dev/tcp tcp_slow_start_initial 2

Varnish Settings Dissected

Here are the most important settings to look out for when deploying Varnish in production.

File Descriptors

Run Varnish under a Solaris project that gives the proxy enough file descriptors to handle the concurrency. If Varnish can not allocate enough file descriptors, it can’t serve the requests.

# Paste into /etc/project
# Run the Application newtask -p highfile 
highfile:101::*:*:process.max-file-descriptor=(basic,32192,deny)

Threads

Give enough idle threads to Varnish so it does not stall on requests. Thread creation is slow and expensive, idle threads are not. Don’t go cheap with threads, allocate a minimum of 200. Modern browsers use 8 concurrent connections by default, meaning Varnish will need 8 threads to handle a single page view.

thread_pool_max=2000  # 2000 max threads per pool
thread_pool_min=50    # 50 min threads per pool
                      # 50 threads x 4 Pools = 200 threads
thread_pools=4        # 4 Pools, Pool per CPU Core.
session_linger=120ms  # How long to keep a thread around
                      # To handle further requests.

Once again I have been blind sided by yet another conservative out-of-the-box setting. IPFilter is tuned way too conservative with it’s state table size.

Here is how you can tell if your hitting any issues, run ipfstat and check for lost packets.

victori@opensolaris:~# ipfstat | grep lost
fragment state(in):     kept 0  lost 0  not fragmented 0
fragment state(out):    kept 0  lost 0  not fragmented 0
packet state(in):       kept 798        lost 100
packet state(out):      kept 612        lost 234

Notice that the in and out lost state lines have a non-zero value. This means IPFilter has been dropping client connections, bummer.

The default settings are quite conservative.

You need to shutdown IPFilter and apply larger table size limits.

Lets confirm that it works.

Awesome, now all we need to do is enable IPfilter and no more lost packets.

To make this persistent across reboots edit ipf.conf

Then update the contents

This can be applied to any OS that uses IPFilter.

I am in the process of evaluating which option to choose for a new production deployment of a Sinatra application.

Pros and Cons of the implementations:

JRuby Stack:

Pros:

Cons:

MRI Ruby Stack:

Pros:

Cons:

These tests are run against a real-world application that is soon to be released, not some dummy “hello world” application.

Application Background:

Sinatra / HAML templates (not compiled, rendered per request) / CouchDB / R18N Translation

Server Specifications:

Here are the results. I have taken the best time out of 10 runs, giving enough time for the JDK to warmup and passenger to load all the children. The results are clipped for brevity.

Benchmark command:

JRuby Results:

MRI Results:

Conclusions and final thoughts:

Seems like MRI Ruby has a 39% performance advantage on JRuby executing my application. I am still a bit skeptical if MRI Ruby would still win out in production when it turns into a long running process marathon with varied traffic patterns. At the end of the day the JVM currently has the edge in garbage collection on MRI Ruby, so in “theory” JRuby should be the better choice. This is all a hypothetical guesstimate[sic] on my behalf. I will most likely end up trying both variants in production and see which works best.

One of our articles on fabulously40 went viral on the tagged.com which is one of the largest social networks in the alexa top 100. The viral aspect was quite apparent when the bandwidth sky rocketed to 30Mb/sec of sustained traffic. We were pushing over 60gigs of image data per day! We broke our bandwidth quota in just 3 days. Granted, this was toward the end of the billing cycle for that month.

Facing overdraft charges for bandwidth I decided to re-encode the images with ImageMagick. I converted the images to a “lower quality” compression for a file savings of 71%. Swapping out the images with the newer smaller images dropped the bandwidth by 50% Awesome.

Spot the difference where I swapped out the images?

I have recently blogged about swapping malloc implementations for the JVM to help boost multi-threaded performance. Well there is yet another malloc implementation that solaris comes with that is optimized for single threaded performance; bsdmalloc. I just recently switched our perl interpreter to use bsdmalloc and got 33% faster performance with our perlbal proxy.

You can try out multiple malloc implementations by setting LD_PRELOAD environment variable.

LD_PRELOAD="/usr/lib/libbsdmalloc.so" perl somecode.pl

So here is the rule of thumb for which malloc implementation to use for your application.

libumem = For multithreaded applications. umem avoids thread heap contention and is highly optimized for multi-threaded applications.

bsdmalloc = For single threaded applications. PHP/Perl/Python and Ruby will fall into this category.

Applying the right malloc implementation to your resource intensive application can see a nice performance benefit.

Apparently Solaris comes with some crummy settings for web hosting. Here are the settings I have used to improve our web performance at our service.

victori@fab40:/etc/rc2.d# netstat -sP tcp | grep -i drop
	tcpTimRetransDrop   =  6029	tcpTimKeepalive     =  2467
	tcpListenDrop       = 27327	tcpListenDropQ0     =     0
	tcpHalfOpenDrop     =     0	tcpOutSackRetrans   = 99988

If tcpListenDrop is above 0, you have more connections than the system can handle with the default settings. Increasing tcp_conn_req_max_q accordingly should fix the issue. Raise the number incrementally until tcpListenDrop stops gradually increasing.

The tcp_conn_req_max_q default is 1024.

/usr/sbin/ndd -set /dev/tcp tcp_conn_req_max_q 8192
/usr/sbin/ndd -set /dev/tcp tcp_conn_req_max_q0 8192

Lower the anonymous port range to support the larger connection queue that was defined.

/usr/sbin/ndd -set /dev/tcp tcp_smallest_anon_port 2048

Up the buffer size for transmissions, to you know….. actually make use of that 100mbit connection?

/usr/sbin/ndd -set /dev/tcp tcp_xmit_hiwat 1048576
/usr/sbin/ndd -set /dev/tcp tcp_recv_hiwat 1048576
/usr/sbin/ndd -set /dev/tcp tcp_max_buf 2097152

To persist these settings across a reboot just write out the contents to /etc/rc2.d/S99netoptimize bash file

I was told Solaris was configured out of the box for today’s hardware? wtf?

OpenSolaris uses a single-threaded malloc by default for all applications. The JDK that is compiled for Solaris fails to be linked against mtmalloc or the newer umem malloc implementation that is multithread optimized. In a multithreaded application using a single threaded malloc can degrade performance. As memory is being allocated concurrently in multiple threads, all the threads must wait in a queue while malloc() handles one request at a time, this is called heap contention. To get around this contention point you can force the JDK to use the umem malloc. 

LD_PRELOAD=/usr/lib/libumem.so /opt/jdk1.7.0/bin/java start.jar
or
LD_PRELOAD=/usr/lib/libmtmalloc.so /opt/jdk1.7.0/bin/java start.jar

This simple fix has really improved performance on our web service fabulously40. The application went from serving 120req/sec uncached to 170req/sec. Not bad no? 

This also works wonders for mysql and varnish, two applications that really put those threads to use.  We have dropped 100ms in response time with varnish by just using umem for the malloc implementation.

I am not exactly sure why this isn’t documented but nginx as of 0.7.x supports event ports

This is a huge performance win for Solaris. Nginx can avoid the 0(n) file descriptor problem with event ports support.

To enable event ports add this to your nginx.conf

events {
    use eventport;
}

Here is our performance-proven configuration that we use on fabulously40

The follow configuration will help you survive massive traffic with nginx. We have served 4.4 million requests in a 4 hour time frame with no issues. That is 305req/sec.

worker_processes  8;
worker_rlimit_nofile 10240;
events {
    worker_connections  8024;
    use eventport;
}
http {
    keepalive_timeout  20;
    server_names_hash_bucket_size 64;
    sendfile        on;
    tcp_nopush     on;
    client_max_body_size 150m;

    gzip  on;
    gzip_comp_level  5;
    gzip_vary       on;
    gzip_proxied    any;
    gzip_types      text text/plain text/css text/xml application/xml text/javascript text/html application/x-javascript;
}