Catalyst vs Rails Round Two

There seems to be some interest Catalyst vs Rails vs Django benchmark. The older benchmark is quite old, it was done in 2007. A lot has changed since then. I am re-running the numbers once again to see what has changed. This time around the hardware is faster and the benchmark is slightly more simple. I am just stress testing the controller response performance of the two frameworks.

Benchmark System:
Quad Core Xeon x5355 @ 2.66GHz,8 Gigs Ram,OpenSolaris SNV98

Quick Summary:
Catalyst 5.8/Perl 5.10: 611.78req/sec (Single Process,bsdmalloc)
Catalyst 5.8/Perl 5.10: 1485.53req/sec (Multi Process,bsdmalloc)
Rails 2.3.2/MRI Ruby 1.8.7: 259.93req/sec (Single Process,bsdmalloc)
Rails 2.3.2/JRuby 1.3-dev: 311.71req/sec (Single-Threaded,bsdmalloc)
Rails 2.3.2/JRuby 1.3-dev: 992.32req/sec (Multi-Threaded,libumem)
Rails 2.3.2/MRI Ruby 1.9.1: 603.92req/sec (Single Process,bsdmalloc)

Jump to conclusion….

Catalyst 5.8 / Perl 5.10
Compiled: SUNCC -xO5 -xipo -fast -xtarget=native


# ab -n1000 -c100 http://somedomain:3000/
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Finished 1000 requests


Server Software:        
Server Hostname:        somedomain
Server Port:            3000

Document Path:          /
Document Length:        11 bytes

Concurrency Level:      100
Time taken for tests:   0.673159 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      159000 bytes
HTML transferred:       11000 bytes
Requests per second:    1485.53 [#/sec] (mean)
Time per request:       67.316 [ms] (mean)
Time per request:       0.673 [ms] (mean, across all concurrent requests)
Transfer rate:          230.26 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   2.3      0      35
Processing:    15   62  10.8     62     103
Waiting:       15   59  11.6     61      98
Total:         15   62  10.6     63     103

Percentage of the requests served within a certain time (ms)
  50%     63
  66%     66
  75%     69
  80%     71
  90%     74
  95%     76
  98%     86
  99%     91
 100%    103 (longest request)

Rails 2.3.2 / Ruby 1.8.7
Compiled: SUNCC -xO5 -xipo -fast -xtarget=native



# ab -n1000 -c100 http://somedomain:3000/main/index
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking somedomain (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        Mongrel
Server Hostname:        somedomain
Server Port:            3000

Document Path:          /main/index
Document Length:        11 bytes

Concurrency Level:      100
Time taken for tests:   3.847 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      290003 bytes
HTML transferred:       11000 bytes
Requests per second:    259.93 [#/sec] (mean)
Time per request:       384.718 [ms] (mean)
Time per request:       3.847 [ms] (mean, across all concurrent requests)
Transfer rate:          73.61 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   1.1      0      16
Processing:    10  369  65.2    389     428
Waiting:        9  368  65.3    388     427
Total:         10  369  65.2    390     428

Percentage of the requests served within a certain time (ms)
  50%    390
  66%    396
  75%    398
  80%    400
  90%    404
  95%    407
  98%    413
  99%    417
 100%    428 (longest request)

Rails 2.3.2 / JRuby 1.3-dev build 6586 (Multi-Threaded), libumem
Platform: JDK7 B56

# ab -n1000 -c100 http://somedomain.com:3000/main/index
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking somedomain.com (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        
Server Hostname:        somedomain.com
Server Port:            3000

Document Path:          /main/index
Document Length:        11 bytes

Concurrency Level:      100
Time taken for tests:   1.008 seconds
Complete requests:      1000
Failed requests:        1
   (Connect: 0, Receive: 0, Length: 1, Exceptions: 0)
Write errors:           0
Non-2xx responses:      1
Total transferred:      253875 bytes
HTML transferred:       11936 bytes
Requests per second:    992.32 [#/sec] (mean)
Time per request:       100.773 [ms] (mean)
Time per request:       1.008 [ms] (mean, across all concurrent requests)
Transfer rate:          246.02 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   1.7      0      13
Processing:    10   79  18.8     80     122
Waiting:        9   79  18.8     79     122
Total:         10   80  18.9     80     122

Percentage of the requests served within a certain time (ms)
  50%     80
  66%     88
  75%     94
  80%     98
  90%    102
  95%    108
  98%    113
  99%    114
 100%    122 (longest request)

Rails 2.3.2 / JRuby 1.3-dev build 6586 (Single-Threaded)
Platform: JDK7 B56


# ab -n1000 -c100 http://somedomain:3000/main/index
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking somedomain (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        
Server Hostname:        somedomain
Server Port:            3000

Document Path:          /main/index
Document Length:        11 bytes

Concurrency Level:      100
Time taken for tests:   3.208 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      253000 bytes
HTML transferred:       11000 bytes
Requests per second:    311.71 [#/sec] (mean)
Time per request:       320.810 [ms] (mean)
Time per request:       3.208 [ms] (mean, across all concurrent requests)
Transfer rate:          77.01 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   2.5      0      31
Processing:    37  304  56.4    318     350
Waiting:       36  304  56.5    318     349
Total:         37  305  56.5    318     352

Percentage of the requests served within a certain time (ms)
  50%    318
  66%    326
  75%    330
  80%    332
  90%    336
  95%    341
  98%    345
  99%    348
 100%    352 (longest request)

Rails 2.3.2 / Ruby 1.9.1
Compiled: GCC -O3 -fomit-frame-pointer (SunCC failed to compile)



# ab -n1000 -c100 http://somedomain:3000/main/index
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking somedomain (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        thin
Server Hostname:        somedomain
Server Port:            3000

Document Path:          /main/index
Document Length:        11 bytes

Concurrency Level:      100
Time taken for tests:   1.656 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      267001 bytes
HTML transferred:       11000 bytes
Requests per second:    603.92 [#/sec] (mean)
Time per request:       165.585 [ms] (mean)
Time per request:       1.656 [ms] (mean, across all concurrent requests)
Transfer rate:          157.47 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.6      0       8
Processing:    28  160  39.0    187     222
Waiting:       13  145  38.5    143     209
Total:         28  160  39.0    187     222

Percentage of the requests served within a certain time (ms)
  50%    187
  66%    190
  75%    191
  80%    191
  90%    196
  95%    201
  98%    221
  99%    221
 100%    222 (longest request)

Conclusion

Seems like Catalyst has the edge in controller performance compared to Rails on MRI Ruby 1.8.7. Catalyst's controller processing is 135% faster than Rails in single process performance and 471% faster as a forking multi process. It is nice to see that the Catalyst team addressed the controller performance short comings of the earlier versions of Catalyst. Like any benchmark take it with a grain of salt. In a real application your data access layer will most likely be the bottle neck.

Rails 2.3.2 under JRuby with threading enabled ran 283% faster than with MRI Ruby 1.8.7. I am anxiously waiting on JDK7 B57 with invoke dynamic support, this should help push JRuby's performance even further. I guess I know what deployment option I will choose when deploying Rails.

Pick your poison, both frameworks provide excellent controller response performance. Keep in mind scaling is all about architecture and not how fast your controller's responses are. That said, having an efficient framework does help 😉

11 comments so far

Ian Kallen says:

May 1, 2009 at 8:56 pm

Wow, maybe rails isn’t ruby’s killer app, it’s jruby.
Nateware says:

May 13, 2009 at 8:05 pm

Yeah, but this benchmark misses the point that Catalyst still sucks.

We tried to develop in it, since mod_perl screams and I contributed FormBuilder and SQL::Abstract which are heavily used within Catalyst. It was a horrible experience which I’ve tried to wipe from my memory at all costs.

When we switched to Rails, my stress level decreased by dramatically, and our app performs quite well.

The database is always the bottleneck anyways.
john napiorkowski says:

May 28, 2009 at 10:03 am

I’m sorry you had such a bad experience with Catalyst, however you may not have dug so deeply since SQL::Abstract is used not in Catalyst but in DBIx::Class (a pure Perl ORM) and FormBuilder is a completely separate project. You’re confusion is reasonable, since many people contribute to both DBIx::Class and Catalyst and a lot of the examples floating around do use both.

Formbuilder is probably not the recommended system at this time.

Since I don’t know what you were trying to do, I really can’t offer any suggestions. I have the opposite experience from you. I pulled my hair out trying to make Rails do what I needed. I find Rails seems to be fine for brand new project with no legacy code, since you have to restrict yourself very heavily to the Rails Cool Aid to get the benefits. But when I need to integrate legacy databases, or have databases with multi field primary / foreign keys, etc. I just didn’t enjoy it. Again, it might be a personality thing as well, programming languages fit people like gloves.
Nilson Santos F. Jr. says:

July 1, 2009 at 9:31 am

I think Nateware comment is extremely unhelpful to the passerby as he lists his frustrations with two other technologies *unrelated* to Catalyst.

Neither FormBuilder nor SQL::Abstract are parts of Catalyst. As a matter of fact, FormBuilder isn’t even a recommended option at this point.
Radu B. says:

July 17, 2009 at 9:02 am

Where is django in the new benchmark? 🙂
I really think it could be very interesting to know the difference…
Tex says:

August 24, 2009 at 3:02 am

Hi, very interesting article.

I have a question: when you refer to JRuby Multi-Threaded what do you mean ? config.thread_safe! enabled on config/environments/production.rb or something else ?

I’m wondering how to handle concurrent requests on rails 2.3.x using JRuby on Glassfish Gem, using Mongrel Java and so on but without results…

Example:

rails concurrent
cd concurrent
script/generate controller test test

edit app/controllers/test.rb

class TestController < ApplicationController def test sleep 10 @value = Time.now end end

edit app/views/test/test.test.html.erb

edit config/database.yml

production: adapter: jdbcsqlite3 database: db/production.sqlite3 pool: 5 timeout: 5000

Now launch glassfish gem in production mode:

jruby -S glassfish -e production -n 4 –runtimes-min 4 –runtimes-max 4

Now try (concurrently) get 4 times the page:

http://localhost:3000/test/test

I receive the results queued (the last result after 40 seconds) this happens for glassfish gem, webrick, mongrel…

How can I handle concurrent requests under jruby/rails ?
victori says:

August 31, 2009 at 1:24 pm

@Ted

Sorry for the late response…

1. config.threadsafe! is enabled and jruby-rack is configured for a single runtime.

2. I use jetty as our production and test container. It handles requests concurrently due to config.threadsafe being enabled.

Hope that helps.
Caleb Cushing ( xenoterracide ) says:

October 24, 2009 at 3:10 am

Any Chance you’d throw some benchmarks of django up? and maybe one of the php frameworks?
Mathieu says:

November 13, 2009 at 1:11 pm

how do you deploy your applications ?
mod_perl for catalyst ?
passenger for rails ?

and you should not compile ruby with -O3, -OS is better. there are some articles about that around the net.

and I believe MRI != 1.9, you can’t call 1.9 MRI.
davewood says:

November 9, 2010 at 2:30 am

Currently fastcgi with apache seems to be the most recommended deployment method for catalyst.

Alternatives:
– nginx
– lighttpd
– apache(mode_perl, fastcgi)
– IIS
– LiteSpeed
– Starman

http://wiki.catalystframework.org/wiki/deployment
c9s says:

June 21, 2011 at 4:23 am

by using Perl5.10 + Feersum + Plack,

you can reach more than 15000+ requests per sec.

Letsgetdugg