Letsgetdugg» benchmark

Nginx Passenger vs JRuby on Jetty

victori — Mon, 31 Aug 2009 21:16:39 +0000

I am in the process of evaluating which option to choose for a new production deployment of a Sinatra application.

Pros and Cons of the implementations:

JRuby Stack:

Pros:

• Fast for serial execution once warmed up.
• Multi-threaded, easy to scale with spiked traffic / shared resources.

Cons:

• Slow warm up time, app restarts can really hurt.
• Single process is a single point of failure.

MRI Ruby Stack:

Pros:

• Starts off fast, with no warmup time.
• Scaled via processes, no single point of failure.

Cons:

• Slower than JRuby in serial execution.
• Single Process, no shared resources (Possibly using more memory over time).

These tests are run against a real-world application that is soon to be released, not some dummy “hello world” application.

Application Background:

Sinatra / HAML templates (not compiled, rendered per request) / CouchDB / R18N Translation

Server Specifications:

OS: OpenSolaris – SNV98
Hardware: 8Gig / Quad Core Xeon x5355

MRI Stack:
Ruby 1.8.7 (2008-08-11 patchlevel 72)
Nginx Passenger 2.2.4
Passenger Config: passenger_max_pool_size 8, passenger_use_global_queue on

Java Stack:
JRuby 1.3.1 (ruby 1.8.6p287) (2009-06-15 6586)
Jetty-6.1.15
JDK Flags: -server -Xverify:none -XX:MaxPermSize=96m -XX:+AggressiveOpts -Xss128k -Xms256m -Xmx384m -XX:+UseParallelGC -XX:+UseParallelOldGC
JDK 1.7.0 b67

Here are the results. I have taken the best time out of 10 runs, giving enough time for the JDK to warmup and passenger to load all the children. The results are clipped for brevity.

Benchmark command:

ab -n1000 -c10 http://service/

JRuby Results:

Requests per second: 85.97 [#/sec] (mean)
Time per request: 116.316 [ms] (mean)
Time taken for tests: 11.632 seconds
Memory Use After Test: 437M (RSS)

MRI Results:

Requests per second: 118.85 [#/sec] (mean)
Time per request: 84.142 [ms] (mean)
Time taken for tests: 8.414 seconds
Memory Use After Test: 264M (RSS)

Conclusions and final thoughts:

Seems like MRI Ruby has a 39% performance advantage on JRuby executing my application. I am still a bit skeptical if MRI Ruby would still win out in production when it turns into a long running process marathon with varied traffic patterns. At the end of the day the JVM currently has the edge in garbage collection on MRI Ruby, so in “theory” JRuby should be the better choice. This is all a hypothetical guesstimate[sic] on my behalf. I will most likely end up trying both variants in production and see which works best.

Catalyst vs Rails Round Two

victori — Thu, 30 Apr 2009 20:23:34 +0000

There seems to be some interest Catalyst vs Rails vs Django benchmark. The older benchmark is quite old, it was done in 2007. A lot has changed since then. I am re-running the numbers once again to see what has changed. This time around the hardware is faster and the benchmark is slightly more simple. I am just stress testing the controller response performance of the two frameworks.

Benchmark System:
Quad Core Xeon x5355 @ 2.66GHz,8 Gigs Ram,OpenSolaris SNV98

Quick Summary:
Catalyst 5.8/Perl 5.10: 611.78req/sec (Single Process,bsdmalloc)
Catalyst 5.8/Perl 5.10: 1485.53req/sec (Multi Process,bsdmalloc)
Rails 2.3.2/MRI Ruby 1.8.7: 259.93req/sec (Single Process,bsdmalloc)
Rails 2.3.2/JRuby 1.3-dev: 311.71req/sec (Single-Threaded,bsdmalloc)
Rails 2.3.2/JRuby 1.3-dev: 992.32req/sec (Multi-Threaded,libumem)
Rails 2.3.2/MRI Ruby 1.9.1: 603.92req/sec (Single Process,bsdmalloc)

Jump to conclusion….

Catalyst 5.8 / Perl 5.10
Compiled: SUNCC -xO5 -xipo -fast -xtarget=native

# ab -n1000 -c100 http://somedomain:3000/
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Finished 1000 requests

Server Software:
Server Hostname:        somedomain
Server Port:            3000

Document Path:          /
Document Length:        11 bytes

Concurrency Level:      100
Time taken for tests:   0.673159 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      159000 bytes
HTML transferred:       11000 bytes
Requests per second:    1485.53 [#/sec] (mean)
Time per request:       67.316 [ms] (mean)
Time per request:       0.673 [ms] (mean, across all concurrent requests)
Transfer rate:          230.26 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   2.3      0      35
Processing:    15   62  10.8     62     103
Waiting:       15   59  11.6     61      98
Total:         15   62  10.6     63     103

Percentage of the requests served within a certain time (ms)
  50%     63
  66%     66
  75%     69
  80%     71
  90%     74
  95%     76
  98%     86
  99%     91
 100%    103 (longest request)

Rails 2.3.2 / Ruby 1.8.7
Compiled: SUNCC -xO5 -xipo -fast -xtarget=native

# ab -n1000 -c100 http://somedomain:3000/main/index
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking somedomain (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests

Server Software:        Mongrel
Server Hostname:        somedomain
Server Port:            3000

Document Path:          /main/index
Document Length:        11 bytes

Concurrency Level:      100
Time taken for tests:   3.847 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      290003 bytes
HTML transferred:       11000 bytes
Requests per second:    259.93 [#/sec] (mean)
Time per request:       384.718 [ms] (mean)
Time per request:       3.847 [ms] (mean, across all concurrent requests)
Transfer rate:          73.61 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   1.1      0      16
Processing:    10  369  65.2    389     428
Waiting:        9  368  65.3    388     427
Total:         10  369  65.2    390     428

Percentage of the requests served within a certain time (ms)
  50%    390
  66%    396
  75%    398
  80%    400
  90%    404
  95%    407
  98%    413
  99%    417
 100%    428 (longest request)

Rails 2.3.2 / JRuby 1.3-dev build 6586 (Multi-Threaded), libumem
Platform: JDK7 B56

# ab -n1000 -c100 http://somedomain.com:3000/main/index
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking somedomain.com (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests

Server Software:
Server Hostname:        somedomain.com
Server Port:            3000

Document Path:          /main/index
Document Length:        11 bytes

Concurrency Level:      100
Time taken for tests:   1.008 seconds
Complete requests:      1000
Failed requests:        1
   (Connect: 0, Receive: 0, Length: 1, Exceptions: 0)
Write errors:           0
Non-2xx responses:      1
Total transferred:      253875 bytes
HTML transferred:       11936 bytes
Requests per second:    992.32 [#/sec] (mean)
Time per request:       100.773 [ms] (mean)
Time per request:       1.008 [ms] (mean, across all concurrent requests)
Transfer rate:          246.02 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   1.7      0      13
Processing:    10   79  18.8     80     122
Waiting:        9   79  18.8     79     122
Total:         10   80  18.9     80     122

Percentage of the requests served within a certain time (ms)
  50%     80
  66%     88
  75%     94
  80%     98
  90%    102
  95%    108
  98%    113
  99%    114
 100%    122 (longest request)

Rails 2.3.2 / JRuby 1.3-dev build 6586 (Single-Threaded)
Platform: JDK7 B56

# ab -n1000 -c100 http://somedomain:3000/main/index
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking somedomain (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests

Server Software:
Server Hostname:        somedomain
Server Port:            3000

Document Path:          /main/index
Document Length:        11 bytes

Concurrency Level:      100
Time taken for tests:   3.208 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      253000 bytes
HTML transferred:       11000 bytes
Requests per second:    311.71 [#/sec] (mean)
Time per request:       320.810 [ms] (mean)
Time per request:       3.208 [ms] (mean, across all concurrent requests)
Transfer rate:          77.01 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   2.5      0      31
Processing:    37  304  56.4    318     350
Waiting:       36  304  56.5    318     349
Total:         37  305  56.5    318     352

Percentage of the requests served within a certain time (ms)
  50%    318
  66%    326
  75%    330
  80%    332
  90%    336
  95%    341
  98%    345
  99%    348
 100%    352 (longest request)

Rails 2.3.2 / Ruby 1.9.1
Compiled: GCC -O3 -fomit-frame-pointer (SunCC failed to compile)

# ab -n1000 -c100 http://somedomain:3000/main/index
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking somedomain (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests

Server Software:        thin
Server Hostname:        somedomain
Server Port:            3000

Document Path:          /main/index
Document Length:        11 bytes

Concurrency Level:      100
Time taken for tests:   1.656 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      267001 bytes
HTML transferred:       11000 bytes
Requests per second:    603.92 [#/sec] (mean)
Time per request:       165.585 [ms] (mean)
Time per request:       1.656 [ms] (mean, across all concurrent requests)
Transfer rate:          157.47 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.6      0       8
Processing:    28  160  39.0    187     222
Waiting:       13  145  38.5    143     209
Total:         28  160  39.0    187     222

Percentage of the requests served within a certain time (ms)
  50%    187
  66%    190
  75%    191
  80%    191
  90%    196
  95%    201
  98%    221
  99%    221
 100%    222 (longest request)

Conclusion

Seems like Catalyst has the edge in controller performance compared to Rails on MRI Ruby 1.8.7. Catalyst’s controller processing is 135% faster than Rails in single process performance and 471% faster as a forking multi process. It is nice to see that the Catalyst team addressed the controller performance short comings of the earlier versions of Catalyst. Like any benchmark take it with a grain of salt. In a real application your data access layer will most likely be the bottle neck.

Rails 2.3.2 under JRuby with threading enabled ran 283% faster than with MRI Ruby 1.8.7. I am anxiously waiting on JDK7 B57 with invoke dynamic support, this should help push JRuby’s performance even further. I guess I know what deployment option I will choose when deploying Rails.

Pick your poison, both frameworks provide excellent controller response performance. Keep in mind scaling is all about architecture and not how fast your controller’s responses are. That said, having an efficient framework does help

Not all malloc implementations are created equal

victori — Thu, 30 Apr 2009 07:25:54 +0000

I have recently blogged about swapping malloc implementations for the JVM to help boost multi-threaded performance. Well there is yet another malloc implementation that solaris comes with that is optimized for single threaded performance; bsdmalloc. I just recently switched our perl interpreter to use bsdmalloc and got 33% faster performance with our perlbal proxy.

You can try out multiple malloc implementations by setting LD_PRELOAD environment variable.

LD_PRELOAD="/usr/lib/libbsdmalloc.so" perl somecode.pl

So here is the rule of thumb for which malloc implementation to use for your application.

libumem = For multithreaded applications. umem avoids thread heap contention and is highly optimized for multi-threaded applications.

bsdmalloc = For single threaded applications. PHP/Perl/Python and Ruby will fall into this category.

Applying the right malloc implementation to your resource intensive application can see a nice performance benefit.

Ruby scaling up to multiple CPUs

victori — Tue, 28 Apr 2009 20:13:51 +0000

I wrote a quick micro benchmark to test out ruby threads. Apparently ruby can’t make use of multiple CPUs with it’s threading implementation. I guess you have to resort to forking to scale up to multiple cpu cores while using mri ruby. However, there is an alternative solution, just use JRuby. JRuby utilizes all cores when running the benchmark.

Ruby 1.8.7 - Compiled with SunCC SSX0903 (-xO5 -fast -xipo)
Total number of insane floating point divisions in 10 seconds is 5969107

Ruby 1.9.1 / Compiled with GCC 4.3.2 (-O3 -fomit-frame-pointer)
Total number of insane floating point divisions in 10 seconds is 8596894

Ran as: jruby –fast cpuMax.rb
177% increase in performance

JRuby 1.3-dev / JDK7 b56
Total number of insane floating point divisions in 10 seconds is 15915896

Ran as: jruby –fast -J-Djruby.compile.mode=JIT -J-Djruby.jit.threshold=0 -J-server cpuMax.rb
374% increase in performance

JRuby 1.3-dev / JDK7 b56
Total number of insane floating point divisions in 10 seconds is 28334441

Looking at mpstat, I can see the MRI ruby implementation is not utilizing all 4 cores.

Ruby 1.8.7 MRI

CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
  0  828   0   13   428  204  487   26   93   16    0  1903    4   5   0  91
  1 2682   0    3    39    2  280   32   81   12    2  1189   13   2   0  85
  2 1902   0    0    34   11  259   16   57   13    0  1094   11   3   0  86
  3 1017   0    3   192  150  111   34   38    8    0   676   92   2   0   6
CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
  0    2   0    5   422  205  388   11   75    5    0   627    4   1   0  95
  1  161   0   13    20    2  196   11   61    8    0  1405    2   2   0  96
  2  292   0    6    32   15  272   10   57    7    0   700    2   1   0  97
  3    0   0    0   108   65   74   35   28    3    0   346   99   0   0   1

Now here is the JRuby implementation.

CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
  0  229   0  202   755  193 1977  365  329   94    1  2869   90   2   0   8
  1  328   0   86   371    1 1817  226  303  125    0  2809   86   2   0  12
  2  294   0  128   326    0 1771  248  287  109    0  2290   88   2   0  10
  3  320   0  172   402   62 1848  246  241  116    0  2238   86   3   0  11
CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
  0  317   0  297   700  192 2047  323  341  136    1  2819   89   2   0   9
  1  320   0   61   279    2 1611  134  195  130    0  1960   85   2   0  13
  2  288   0  235   379    0 1941  291  299  115    0  2462   87   2   0  11
  3  308   0   78   316   43 1688  142  159  104    0  1706   85   2   0  13

I think I will stick to JRuby for production use.

#!/usr/bin/ruby
require ‘thread’

threads = []

counter = 0

mutex = Mutex.new
4.times do

     threads << Thread.new {

        x=0

        y=0

        time=Time.new
        while 1 do

                if Time.new - time >= 10 then

                        break

                else

                        x=1.00/24000000000.001

                        y+=1

                end

        end

        mutex.synchronize { counter+=y.to_i }

    }

end

threads.each { |t| t.join }
puts "Total number of insane floating point divisions in 10 seconds is "+counter.to_s