Catalyst vs Rails vs Django Cook off
Update: See my newer post that re-evaluates the frameworks. Things have changed since then.
Today I began working on a new project and decided to benchmark Catalyst and Rails for fun. See how my new favorable framework does against Rails. I was a bit shocked at the results though. I guess this is worth mentioning in hope Catalyst can improve in it's Accessor Generation code. So here are the results:
Benchmark System
Celeron 1.8Ghz,1 Gig of Ram,FreeBSD-6
Interpreters:
Ruby – 1.8.5
Perl – 5.8.8
Frameworks:
Catalyst – 5.7003
Rails – 1.1.6
Run as:
Lighttpd: 1.4.13
FCGI:
3 max proc
Benchmarked as:
ab -n 1000 -c 100 http://siteurl.com/
Some background
I specifically turned off sessions and did not use ActiveRecord/DBIC to keep it as fair as possible between the two frameworks. Both frameworks were run under Lighttpd and FCGI. I tried to keep this as apples to apples as possible.
So lets take a look at the results!
Rails:
Server Software: lighttpd/1.4.13
Server Hostname: wansanity.com
Server Port: 9090
Document Path: /main/index
Document Length: 2142 bytes
Concurrency Level: 100
Time taken for tests: 18.261 seconds
Complete requests: 1000
Failed requests: 0
Broken pipe errors: 0
Total transferred: 2296892 bytes
HTML transferred: 2143288 bytes
Requests per second: 54.76 [#/sec] (mean)
Time per request: 1826.10 [ms] (mean)
Time per request: 18.26 [ms] (mean, across all concurrent requests)
Transfer rate: 125.78 [Kbytes/sec] received
Connnection Times (ms)
min mean[+/-sd] median max
Connect: 74 885 1742.9 138 11785
Processing: 172 661 1216.8 173 8195
Waiting: 84 661 1216.8 173 8194
Total: 172 1547 2123.8 330 11893
Percentage of the requests served within a certain time (ms)
50% 330
66% 1354
75% 2786
80% 3106
90% 4297
95% 6279
98% 8216
99% 9285
100% 11893 (last request)
Thats 54 connections / sec which is great. I have seen it peak at 70 connections/sec which is just awesome!
Catalyst:
Server Software: lighttpd/1.4.13
Server Hostname: wansanity.com
Server Port: 80
Document Path: /
Document Length: 2232 bytes
Concurrency Level: 100
Time taken for tests: 43.503 seconds
Complete requests: 1000
Failed requests: 0
Broken pipe errors: 0
Total transferred: 2401300 bytes
HTML transferred: 2238490 bytes
Requests per second: 22.99 [#/sec] (mean)
Time per request: 4350.30 [ms] (mean)
Time per request: 43.50 [ms] (mean, across all concurrent requests)
Transfer rate: 55.20 [Kbytes/sec] received
Connnection Times (ms)
min mean[+/-sd] median max
Connect: 75 322 808.5 93 6028
Processing: 269 3804 851.8 3928 6754
Waiting: 192 3804 851.7 3928 6754
Total: 269 4126 1178.5 4186 10293
Percentage of the requests served within a certain time (ms)
50% 4186
66% 4384
75% 4404
80% 4424
90% 5025
95% 6422
98% 7194
99% 7709
100% 10293 (last request)
22 connections / sec not exactly what I expected from a framework built on top of the fast Perl Interpreter.
Being a bit disappointed with the results, I investigated further.
So here are the perl dprof results.
%Time ExclSec CumulS #Calls sec/call Csec/c Name
0.00 0.605 4.128 1512 0.0004 0.0027 NEXT::AUTOLOAD
0.00 0.373 0.373 25794 0.0000 0.0000 Class::Accessor::Fast::__ANON__
0.00 0.235 0.235 1177 0.0002 0.0002 NEXT::ELSEWHERE::ancestors
0.00 0.211 0.225 1 0.2107 0.2253 YAML::Type::code::BEGIN
0.00 0.184 5.182 86 0.0021 0.0603 Catalyst::Engine::HTTP::_handler
0.00 0.177 0.205 2583 0.0001 0.0001 File::Spec::Unix::canonpath
0.00 0.164 0.309 1942 0.0001 0.0002 File::Spec::Unix::catdir
0.00 0.156 2.408 3201 0.0000 0.0008 Catalyst::Action::__ANON__
0.00 0.134 0.739 73 0.0018 0.0101 base::import
0.00 0.129 0.136 5904 0.0000 0.0000 Class::Data::Inheritable::__ANON__
0.00 0.109 0.814 7 0.0155 0.1163 main::BEGIN
0.00 0.108 0.108 1323 0.0001 0.0001 HTTP::Headers::_header
0.00 0.101 0.116 10 0.0101 0.0116 Template::Parser::BEGIN
0.00 0.101 0.334 11 0.0092 0.0304 Catalyst::Engine::BEGIN
0.00 0.101 0.295 1264 0.0001 0.0002 Path::Class::Dir::stringify
It seems like the main bottleneck in Catalyst 5.7003 is
Next
Jrockway was kind enough to post some new code into Catalyst's trunk for me to try; a new replacement for Next –
C3
Here are the results with the C3 Plugin from Trunk
%Time ExclSec CumulS #Calls sec/call Csec/c Name
0.00 0.211 0.233 1 0.2106 0.2330 YAML::Type::code::BEGIN
0.00 0.135 0.135 8035 0.0000 0.0000 Class::Accessor::Fast::__ANON__
0.00 0.126 0.721 73 0.0017 0.0099 base::import
0.00 0.109 0.116 10 0.0109 0.0116 Template::Parser::BEGIN
0.00 0.108 0.805 7 0.0155 0.1150 main::BEGIN
0.00 0.093 0.106 7 0.0133 0.0152 Catalyst::Engine::HTTP::Restarter:
:Watcher::BEGIN
0.00 0.090 0.105 1023 0.0001 0.0001 File::Spec::Unix::canonpath
0.00 0.085 0.326 11 0.0077 0.0296 Catalyst::Engine::BEGIN
0.00 0.081 0.905 196 0.0004 0.0046 Catalyst::execute
0.00 0.069 0.120 8 0.0087 0.0150 Catalyst::Plugin::Server::XMLRPC::
Request::BEGIN
0.00 0.064 1.639 444 0.0001 0.0037 next::method
0.00 0.061 0.313 32 0.0019 0.0098 Catalyst::BEGIN
0.00 0.054 0.216 7 0.0077 0.0309 Template::Config::load
0.00 0.054 0.189 4 0.0135 0.0473 HTTP::Body::OctetStream::BEGIN
0.00 0.054 0.388 4 0.0135 0.0970 Gambit::BEGIN
So there you have it, the results with the C3 Plugin. It only made a slight difference by pushing the Catalyst benchmark score to 25 connections / sec.
I hope this benchmark can get some changes put into place for Catalyst’s next release.
Conclusion
It seems like Rails is roughly 62% faster than Catalyst at this time. Keep in mind this benchmark does not take into account the ORM performance. This benchmark tests how quick the frameworks themselves dispatch methods and render views.
Also take into consideration when choosing a framework you need to look at the problem at hand. Catalyst can feed off Perl's vast CPAN resource library. Catalyst has features that Rails does not have. Catalyst's DBIC ORM supports multi-column primary keys and can do relationship mapping just by reading the schema! You don't even have to bother writing any
has
many_
belongs
to_ definitions!
I am going to have to take a look into Django see how well it fairs in this benchmark. Perhaps an update on this?
Update Django Results
Server Software: lighttpd/1.4.13
Server Hostname: fab40
Server Port: 9090
Document Path: /
Document Length: 2235 bytes
Concurrency Level: 100
Time taken for tests: 13.643 seconds
Complete requests: 1000
Failed requests: 0
Broken pipe errors: 0
Total transferred: 2409769 bytes
HTML transferred: 2253459 bytes
Requests per second: 73.30 [#/sec] (mean)
Time per request: 1364.30 [ms] (mean)
Time per request: 13.64 [ms] (mean, across all concurrent requests)
Transfer rate: 176.63 [Kbytes/sec] received
Connnection Times (ms)
min mean[+/-sd] median max
Connect: 76 483 1068.1 101 8666
Processing: 190 744 726.3 571 6088
Waiting: 93 744 726.4 572 6088
Total: 190 1227 1414.2 692 9606
Percentage of the requests served within a certain time (ms)
50% 692
66% 972
75% 1209
80% 1445
90% 3282
95% 4020
98% 6414
99% 8113
100% 9606 (last request)
72 connections / sec! Amazing and the winner!
And anyone that disagrees with this can go ahead and look at the
code for all three projects
I have the least experience with django for your information
mst Please don’t kill me’
Many thanks go out to jrockway to helping me point out the root cause of the bottleneck in Catalyst.
I’m curious why you used such dated versions of … well … everything. Is that just what you had available under FreeBSD 6?
This post is actually from 2007. I had to convert my old posts from typeface to wordpress.
can you benchmark merb? I am sure merb is the winner 🙂
See http://letsgetdugg.com/2009/04/30/catalyst-vs-rails-round-two/ for new results.
Things have changed *a lot* since this article was written, Catalyst was completely ported to Moose OO system for Perl and is at version 5.80024 now and leverage all the goodies that comes from the Moose ecosystem as well as CPAN. I’m sure that Django and Rails have had there improvements as well so I feel these results are totally obselete now and another comparison is due
by using Perl5.10 + Feersum + Plack,
you can reach more than 15000+ requests per sec.
It’s going to be finish of mine day, except before ending I am reading this impressive piece of writing to increase my experience.
http://www.amirart.com/informat.html
When some one searches for his required thing, thus he/she
wishes to be available that in detail, thus that thing is maintained over here.
The project contractor was corrupted to its needs and requirements.
This can be used for contractor demolition at Lagonda Drive,
and energy friendly to keep in mind. Under reporting can take advantage
of the people that provide demolition, excavation, grading and
hauling contractors also have to make a decision. Underfloor heating, air sealing is
a heavy price for your renovations in a limited section of
the Contractor Open House, Rep. The basement flood damage, do
they normally involve the collection of e-mails of foreigners from the company handled.
Hi there! Do you know if they make any plugins to assist with
Search Engine Optimization? I’m trying to get my blog to rank for some targeted keywords but I’m not
seeing very good gains. If you know of any please share. Thank you!
But with competition intensifying in the field of
public officials involved unlicensed contractors in the first contractor
they hire independent contractors.
This is very much care you may finally end up being a contractor that you school
bus should get estimates.
Hi there to every one, because I am really keen of reading this webpage’s post to
be updated regularly. It consists of pleasant material.
Lastly school bus you need roofing you can make your decision has to be carefully set in your home you have to
meet President Raul Castro in an expert business to clean up.
The government pays people whose pension plans it handled.
Repairing the parts he was employed by a contractor, you might find out the length of service that you can about your credit card.
Howdy! I could have sworn I’ve been to this blog before but after browsing
through a few of the articles I realized it’s new to me.
Nonetheless, I’m certainly happy I came across it and I’ll be bookmarking it
and checking back regularly!
Hello, i believe that i noticed you visited my site thus i got here to go back the choose?.I’m attempting to in finding issues to improve my site!I assume its ok to make use of some of your ideas!!
If you have been feeling tired and rundown lately, and seem to be gaining weight for
no apparent reason, then it is time for a change. Safety is also assured when taking Adiphene because of its natural ingredients, which means that serious side effects are
unlikely to happen if taken as directed. Assuming that
you aren’t a super-taught individual, you will have
some major snags adding on control over your zealous urges towards consuming distinctive sustenances.
Such are the ingredients of Adiphene that its
one finest selling level is the shortage of dangerous uncomfortable side effects.
Perhaps the main reason behind this though is that you have developed
a dependence on food. With more time now passed, those rates would
be even higher as obesity is still on the rise.
Thankfulness to my father who shared with me on the topic of this weblog,
this blog is really awesome.
If you have been feeling tired and rundown lately, and seem to be
gaining weight for no apparent reason, then it is time for a change.
Adiphene is the newest and fastest weight reduction supplement in market.
Sonnie Mc – Lemore is a health and fitness
blog owner.
You wont find Adiphene in your native pharmacy or some huge store like Walmart or Walgreens.
This means thay everybody who makes use of Adiphene should shed pounds.
The science behind Adiphene consists of 12 of probably the most power fat
preventing ingredients known.
You should be a part of a contest for one of the most useful sites on the web.
I’m going to highly recommend this web site!
They are a very good way to promote business service.
However Yahoo marketing and other pay per click search engines offer
some excellent traffic at a more reasonable price.
To report stolen email addresses at Yahoo, click Yahoo.
The best way to take bodyweight off and keep it off is through diet,
exercising aerobically and free weights education. Needless to say,
the right equipment is not the only criteria that will help you reach your objective, making the decision to get started today almost
certainly will guarantee that you reach any and all of
your body building dreams. However, if strength, power, and muscle
mass are one to your goals, then consider the Power Block Elite Set or even the Power Block Pro Rexan models.