Today, I decided to do a little benchmark to test various deployment combinations. Since Retrospectiva is now *fully Ruby 1.9 compatible*, I thought I will see how much performance benefit the new "Ruby":http://ruby-lang.org version is able to deliver. Additionally, the guys from "Phusion":http://modrails.com have released a new version of their Passenger including support fort the "Nginx":http://wiki.Nginx.org web server. Does that sound like a winning combination? Let's see! h2. The Benchmark My test setup was fairly simple. I installed Apache and Nginx on my Ubuntu 8.10 with default configurations and ran each *ab* command three times for each test (to warm up the server). Software: * Apache 2.2.9 (mpm-worker) * Nginx 0.6.36 * Passenger 2.2.0 * Ruby 1.8.7 (from DEB) * Ruby 1.9.1 (source-compiled) The first URL (/search) doesn't involve much DB work load, good for testing ruby performance + web server delivery. The second URL (/ticket/176) simply hits a very typical application page (a ticket). Please note that this benchmark is not really representative for the general performance, since it was run on my very low-powered (32-bit) workstation with only a minimal MySQL setup and the Gnome desktop running. Still, it should give a good indication what to expect from different deployment combinations. h4. Apache2 / Ruby 1.8.7 {{{ $ ab -n 100 -c 5 http://localhost/search # 5 concurrent requests / 100 total Time taken for tests: 8.157 seconds Total transferred: 456200 bytes HTML transferred: 391100 bytes Requests per second: 12.26 [#/sec] (mean) Time per request: 407.845 [ms] (mean) Time per request: 81.569 [ms] (mean, across all concurrent requests) Transfer rate: 54.62 [Kbytes/sec] received $ ab -n 100 -c 5 http://localhost/tickets/176 # 5 concurrent requests / 100 total Time taken for tests: 29.044 seconds Total transferred: 3651760 bytes HTML transferred: 3586465 bytes Requests per second: 3.44 [#/sec] (mean) Time per request: 1452.188 [ms] (mean) Time per request: 290.438 [ms] (mean, across all concurrent requests) Transfer rate: 122.79 [Kbytes/sec] received }}} h4. Nginx / Ruby 1.8.7 {{{ $ ab -n 100 -c 5 http://localhost/search # 5 concurrent requests / 100 total Time taken for tests: 8.368 seconds Total transferred: 450700 bytes HTML transferred: 391100 bytes Requests per second: 11.95 [#/sec] (mean) Time per request: 418.406 [ms] (mean) Time per request: 83.681 [ms] (mean, across all concurrent requests) Transfer rate: 52.60 [Kbytes/sec] received $ ab -n 100 -c 5 http://localhost/tickets/176 # 5 concurrent requests / 100 total Time taken for tests: 29.182 seconds Total transferred: 3632465 bytes HTML transferred: 3572668 bytes Requests per second: 3.43 [#/sec] (mean) Time per request: 1459.098 [ms] (mean) Time per request: 291.820 [ms] (mean, across all concurrent requests) Transfer rate: 121.56 [Kbytes/sec] received }}} h4. Apache2 / Ruby 1.9.1 {{{ $ ab -n 100 -c 5 http://localhost/search # 5 concurrent requests / 100 total Time taken for tests: 3.570 seconds Total transferred: 458792 bytes HTML transferred: 391100 bytes Requests per second: 28.01 [#/sec] (mean) Time per request: 178.509 [ms] (mean) Time per request: 35.702 [ms] (mean, across all concurrent requests) Transfer rate: 125.49 [Kbytes/sec] received $ ab -n 100 -c 5 http://localhost/tickets/176 # 5 concurrent requests / 100 total Time taken for tests: 17.285 seconds Total transferred: 3647065 bytes HTML transferred: 3579259 bytes Requests per second: 5.79 [#/sec] (mean) Time per request: 864.257 [ms] (mean) Time per request: 172.851 [ms] (mean, across all concurrent requests) Transfer rate: 206.05 [Kbytes/sec] received $ ab -n 1000 -c 100 http://localhost/search # 100 concurrent requests / 1000 total Time taken for tests: 42.475 seconds Total transferred: 4587990 bytes HTML transferred: 3911000 bytes Requests per second: 23.54 [#/sec] (mean) Time per request: 4247.454 [ms] (mean) Time per request: 42.475 [ms] (mean, across all concurrent requests) Transfer rate: 105.49 [Kbytes/sec] received $ ab -n 1000 -c 400 http://localhost/search # 400 concurrent requests / 1000 total Time taken for tests: 66.294 seconds Total transferred: 4587996 bytes HTML transferred: 3911000 bytes Requests per second: 15.08 [#/sec] (mean) Time per request: 26517.571 [ms] (mean) Time per request: 66.294 [ms] (mean, across all concurrent requests) Transfer rate: 67.58 [Kbytes/sec] received }}} h4. Nginx / Ruby 1.9.1 {{{ $ ab -n 100 -c 5 http://localhost/search # 5 concurrent requests / 100 total Time taken for tests: 3.662 seconds Total transferred: 453296 bytes HTML transferred: 391100 bytes Requests per second: 27.31 [#/sec] (mean) Time per request: 183.085 [ms] (mean) Time per request: 36.617 [ms] (mean, across all concurrent requests) Transfer rate: 120.89 [Kbytes/sec] received $ ab -n 100 -c 5 http://localhost/tickets/176 # 5 concurrent requests / 100 total Time taken for tests: 17.455 seconds Total transferred: 3642453 bytes HTML transferred: 3580139 bytes Requests per second: 5.73 [#/sec] (mean) Time per request: 872.760 [ms] (mean) Time per request: 174.552 [ms] (mean, across all concurrent requests) Transfer rate: 203.78 [Kbytes/sec] received $ ab -n 1000 -c 100 http://localhost/search # 100 concurrent requests / 1000 total Not successful. All tests with more than ~50 concurrent clients resulted in errors. ---------- Uncaught exception in PassengerServer client thread: exception: write() failed: Broken pipe (32) backtrace: in 'void Client::forwardResponse(boost::shared_ptr<Passenger::Application::Session>&, FileDescriptor&)' (HelperServer.cpp:344) in 'bool Client::handleRequest(FileDescriptor&)' (HelperServer.cpp:438) in 'void Client::threadMain()' (HelperServer.cpp:484) *** Exception Errno::EPIPE in Passenger RequestHandler (Broken pipe) (process 900): ---------- I am not sure what exactly was causing them, but I suspect the passenger-Nginx module should be regarded as experimental at this stage. }}} h2. Final Thoughts I must admit, I am quite astonished by the performance of Ruby 1.9. Although I expected it to be faster than 1.8.7, I did not expect a performance boost of this scale. The DEB packaged Ruby versions are also known to be slower than source-compiled ones. Still, in my tests the application was able to handle between 70% and 130% (!!!) more requests in the same amount of time. This is *a highly significant* (and really quite stunning) performance improvement. Surprisingly, my comparison of web servers has also produced a clear winner (and not he one I would have expected). Although I am a big fan of the "Nginx Project":http://wiki.Nginx.org, the currently available version of the Passenger-Nginx-Module (2.2.0) should not be used for production purposes. I had a few problems when I was testing it, most of them I was able to correct myself. Unfortunately, I ran into real problems when dealing with heavy traffic (which is simply not acceptable). Hitting it with many concurrent requests produced errors on the server side and - as a result - many 500 pages. I also expected Nginx to perform (at least a few percent) better than Apache under normal load. Well, I was wrong, quite the opposite was actually true. Apache managed to respond to my requests slightly faster than Nginx and remained _nice and calm_ and successfully served all incoming requests even under heavy load; *a clear winner*! So what next, maybe _JRuby on GlassFish_? 2009-04-18T16:08:25+01:00 24 2 Dimitrij Denissenko dim Douglas Jarquin I was afraid this might happen, but now I know. Apache it is. 2009-04-20T21:08:35+01:00 530 Woody Peterson I've been having that same error under high loads with apache (and Passenger 2.1.2). It seems to be related to the mpm worker settings, as setting more workers helps, although I can't make it go away completely. I was hoping the switch to Passenger 2.2.2 and nginx might make them go away. At least more workers in nginx costs less :) 2009-04-29T21:17:30+01:00 534 Anonymous I have seen a nice speedup, but more on the order of 30%. That source compilation might be responsible for more than you think! 2009-07-02T14:11:23+01:00 541