Brian Knapp <brianknapp / gmail.com> wrote:
> I spent the last few months benchmarking every combination of ruby server,
> runtime, and framework I could find.
> 
> I'm really curious what the ruby community at large thinks of the results.
> 
> http://www.madebymarket.com/blog/dev/ruby-web-benchmark-report.html

Hello, unicorn (and yahns[1] (and Rainbows!)) author here.

Thank you for posting here, I don't comment on commercial sites and
cannot stand forms/images/JS in HTML.

I'm a bit surprised unicorn runs at all :) let alone being faster than
Puma/Thin in some cases.

>    All tests are run on a 2013 Macbook Air 1.3 GHz Intel Core i5 with 8 GB
>    of DDR3 RAM and a 250 GB SSD.

How many CPU cores is that?  Is HT enabled?  Which OS?

I have no idea if unicorn even runs on OSX (or any proprietary OSes).
I doubt yahns does, but yahns supports kqueue on FreeBSD since March
2014 or so.

>    This is not meant to be a good representation of server hardware or
>    anything like that. It's the dev machine I have on hand. It's a great
>    machine, but understand that all of these performance numbers are
>    relative to that level of hardware.

There's not much difference between server and laptop hardware nowadays.

I only develop on GNU/Linux, and occasionally port to FreeBSD as those
are the platforms where deployment usually happens.  Multi-threading
performance might be better on Linux or FreeBSD for Rubinius, too.

>    The fastest framework is Rack. Plain old Rack with no framework at all.
>    It makes intuitive sense that Rack on its own would be the fastest
>    possible, but it”Ēs also worth understanding how much you are giving up
>    by picking a framework like Sinatra or Rails.

Agreed.  I mainly develop and test using Rack.

Memory usage comparisons would also be a good bench, since that's often
a limiting factor for VPS deployments.

unicorn w/4 processes combined     = XXX MB
jruby server(s) w/4 threads        = YYY MB

>    Peak JRuby 1.7.9 performance was 10,159 req/sec using Rack and Unicorn

I do not believe that sentence :P

>    I specifically didn”Ēt test Unicorn with more workers because it would
>    potentially skew the benchmark results in Unicorns favor simply because
>    it would have access to more resources and processing power.

Thin and Puma also support multi-process, too.   So I think they should
be competitive in that config with unicorn on C Ruby.

For JRuby and Rubinius benchmarks, you should specify how many threads
those servers are running (and how many CPUs you have).

> Unicorn Isn”Ēt That Fast

Correct.  Top performance was never the primary goal of unicorn[2].
unicorn is meant to tolerate imperfect (buggy!) applications and give
acceptable, predictable performance using multiple processes to
workaround limitations in C Ruby, frameworks and developer ability.

In contrast, yahns is completely intolerant of fatal application bugs.
It uses multiple processes on C Ruby, but also uses multiple threads,
and one-shot kqueue/epoll so it should work well with Rubinius.
yahns should be as fast as C Ruby or Rubinius allows.

I've long acknowledged unicorn and yahns are limited by Ruby performance.
So I've mainly focused on improving C Ruby performance in the mean time
(and also teaching myself lock-free programming in C).

>      * Unicorn performance doesn't seem very predictable

No surprise with a single worker and non-concurrent, stop-the-world GC.

>      * Unicorn and wrk doesn't seem to benchmark well

I suspect this is because of the lack of persistent connections support
unicorn.  For real-world use, unicorn is designed to be useless without
nginx (or similar) in front of it.

>      * I couldn't get rainbows to work

Rainbows! requires a lot of reading and configuration, so I don't blame
you.  There's too many options, really, and part of the reason I made
yahns[1] into a separate project.


Thank you again for posting here on a non-commercial,
mouseless-user-friendly list so I can respond :)

Note: I never post benchmarks for my own servers:
a) They'll inevitably come off as biased
b) I don't want to make other servers look bad
   (but I am always allowed to make my own servers look bad)

Thus independent evaluations are very welcome :)


[1] http://yahns.yhbt.net/README
    a webserver by a guy who can't even do HTML :P)

[2] I use C if performance is important, but usually Ruby is enough.