Eric,

>  I'm a bit surprised unicorn runs at all :) let alone being faster than
>  Puma/Thin in some cases.

Well at least now there are public benchmarks to prove that unicorn runs.

> How many CPU cores is that?  Is HT enabled?  Which OS?

It's 2 core and I think HT enabled. It's OS X 10.9.4

> I have no idea if unicorn even runs on OSX (or any proprietary OSes).
> I doubt yahns does, but yahns supports kqueue on FreeBSD since March
> 2014 or so.

I haven't tried yahns, but I'll give it a shot and see if it works or not.

> There's not much difference between server and laptop hardware nowadays.

That's a good point, I just didn't want people looking at the raw numbers
and saying, "Hey, my server is way faster than that. This benchmark is
wrong!" Which seems common in most benchmarks. So, I added lots of
disclaimers.

> I only develop on GNU/Linux, and occasionally port to FreeBSD as those
> are the platforms where deployment usually happens.  Multi-threading
> performance might be better on Linux or FreeBSD for Rubinius, too.

Yeah, I will probably run a future test suite against Linux so that it's
closer to a real server setup.

> Agreed.  I mainly develop and test using Rack.

Why don't more programmers write against Rack directly?

> Memory usage comparisons would also be a good bench, since that's often
> a limiting factor for VPS deployments.

> unicorn w/4 processes combined     = XXX MB
> jruby server(s) w/4 threads        = YYY MB

That's a good point and I think I'll add that dimension to a future
benchmark.

> I do not believe that sentence :P

Me either, I fixed it. Thanks for finding that.

> Thin and Puma also support multi-process, too.   So I think they should
> be competitive in that config with unicorn on C Ruby.

> For JRuby and Rubinius benchmarks, you should specify how many threads
> those servers are running (and how many CPUs you have).

That's a very sensible idea. It gets tricky balancing CPU and Memory to
have truly comparable benchmarks. I'll have to see how best to fairly
compare the different servers.


> Correct.  Top performance was never the primary goal of unicorn[2].
> unicorn is meant to tolerate imperfect (buggy!) applications and give
> acceptable, predictable performance using multiple processes to
> workaround limitations in C Ruby, frameworks and developer ability.

Yes, and I totally understand that, but I don't think a lot of people do
the research to understand why Unicorn is different and where the
performance it does have comes from.

> In contrast, yahns is completely intolerant of fatal application bugs.
> It uses multiple processes on C Ruby, but also uses multiple threads,
> and one-shot kqueue/epoll so it should work well with Rubinius.
> yahns should be as fast as C Ruby or Rubinius allows.

Well now I have to try yahns!

> I've long acknowledged unicorn and yahns are limited by Ruby performance.
> So I've mainly focused on improving C Ruby performance in the mean time
> (and also teaching myself lock-free programming in C).

It's great that you are working on improving C Ruby performance. I don't
have the C chops to contribute. Perhaps someday...

> No surprise with a single worker and non-concurrent, stop-the-world GC.

Again, you'd be surprised how few people know about this stuff. Well, maybe
you wouldn't be surprised, but a lot of people read blogs and pick whatever
the flavor of the week is. After the RapGenius blog fight with Heroku, I'm
sure interest in Unicorn jumped a bit. I'm not sure a lot of people
understand when and why you would use Unicorn vs. a different server.


> I suspect this is because of the lack of persistent connections support
> unicorn.  For real-world use, unicorn is designed to be useless without
> nginx (or similar) in front of it.

That seems like a reasonable tradeoff to me.

> Rainbows! requires a lot of reading and configuration, so I don't blame
> you.  There's too many options, really, and part of the reason I made
> yahns[1] into a separate project.

Well that explains the difficulty I had then. :)

> Thank you again for posting here on a non-commercial,
> mouseless-user-friendly list so I can respond :)

You are welcome. I'm just glad to be able to contribute something useful to
the Ruby community at large.


> Note: I never post benchmarks for my own servers:
> a) They'll inevitably come off as biased
> b) I don't want to make other servers look bad
>   (but I am always allowed to make my own servers look bad)

> Thus independent evaluations are very welcome :)

That is a sensible stance to have. All benchmarks are biased, if only by
our lack of knowledge of what we are benchmarking. Some projects go too far
to show that you should use them because of their awesome benchmarks.
Benchmarks are an important dimension, but they aren't the only dimension.


> [1] http://yahns.yhbt.net/README
>    a webserver by a guy who can't even do HTML :P)

I bet your web servers would be faster if you COULD do HTML :P

(Also, nobody can really do HTML)


Eric, I'd also just like to say thank you for all the effort you put in
unicorn, rainbows, and yahns. They are excellent projects and we use
unicorn in production at work every day. At my previous job we used unicorn
to power a site that raised millions of dollars to cure children's cancer,
so your work is doing a lot to help make the world a better place.

-Brian



On Tue, Jul 15, 2014 at 4:14 PM, Eric Wong <normalperson / yhbt.net> wrote:

> Brian Knapp <brianknapp / gmail.com> wrote:
> > I spent the last few months benchmarking every combination of ruby
> server,
> > runtime, and framework I could find.
> >
> > I'm really curious what the ruby community at large thinks of the
> results.
> >
> > http://www.madebymarket.com/blog/dev/ruby-web-benchmark-report.html
>
> Hello, unicorn (and yahns[1] (and Rainbows!)) author here.
>
> Thank you for posting here, I don't comment on commercial sites and
> cannot stand forms/images/JS in HTML.
>
> I'm a bit surprised unicorn runs at all :) let alone being faster than
> Puma/Thin in some cases.
>
> >    All tests are run on a 2013 Macbook Air 1.3 GHz Intel Core i5 with 8
> GB
> >    of DDR3 RAM and a 250 GB SSD.
>
> How many CPU cores is that?  Is HT enabled?  Which OS?
>
> I have no idea if unicorn even runs on OSX (or any proprietary OSes).
> I doubt yahns does, but yahns supports kqueue on FreeBSD since March
> 2014 or so.
>
> >    This is not meant to be a good representation of server hardware or
> >    anything like that. It's the dev machine I have on hand. It's a great
> >    machine, but understand that all of these performance numbers are
> >    relative to that level of hardware.
>
> There's not much difference between server and laptop hardware nowadays.
>
> I only develop on GNU/Linux, and occasionally port to FreeBSD as those
> are the platforms where deployment usually happens.  Multi-threading
> performance might be better on Linux or FreeBSD for Rubinius, too.
>
> >    The fastest framework is Rack. Plain old Rack with no framework at
> all.
> >    It makes intuitive sense that Rack on its own would be the fastest
> >    possible, but it”Ēs also worth understanding how much you aregiving up
> >    by picking a framework like Sinatra or Rails.
>
> Agreed.  I mainly develop and test using Rack.
>
> Memory usage comparisons would also be a good bench, since that's often
> a limiting factor for VPS deployments.
>
> unicorn w/4 processes combined     = XXX MB
> jruby server(s) w/4 threads        = YYY MB
>
> >    Peak JRuby 1.7.9 performance was 10,159 req/sec using Rack and Unicorn
>
> I do not believe that sentence :P
>
> >    I specifically didn”Ēt test Unicorn with more workers becauseit would
> >    potentially skew the benchmark results in Unicorns favor simply
> because
> >    it would have access to more resources and processing power.
>
> Thin and Puma also support multi-process, too.   So I think they should
> be competitive in that config with unicorn on C Ruby.
>
> For JRuby and Rubinius benchmarks, you should specify how many threads
> those servers are running (and how many CPUs you have).
>
> > Unicorn Isn”Ēt That Fast
>
> Correct.  Top performance was never the primary goal of unicorn[2].
> unicorn is meant to tolerate imperfect (buggy!) applications and give
> acceptable, predictable performance using multiple processes to
> workaround limitations in C Ruby, frameworks and developer ability.
>
> In contrast, yahns is completely intolerant of fatal application bugs.
> It uses multiple processes on C Ruby, but also uses multiple threads,
> and one-shot kqueue/epoll so it should work well with Rubinius.
> yahns should be as fast as C Ruby or Rubinius allows.
>
> I've long acknowledged unicorn and yahns are limited by Ruby performance.
> So I've mainly focused on improving C Ruby performance in the mean time
> (and also teaching myself lock-free programming in C).
>
> >      * Unicorn performance doesn't seem very predictable
>
> No surprise with a single worker and non-concurrent, stop-the-world GC.
>
> >      * Unicorn and wrk doesn't seem to benchmark well
>
> I suspect this is because of the lack of persistent connections support
> unicorn.  For real-world use, unicorn is designed to be useless without
> nginx (or similar) in front of it.
>
> >      * I couldn't get rainbows to work
>
> Rainbows! requires a lot of reading and configuration, so I don't blame
> you.  There's too many options, really, and part of the reason I made
> yahns[1] into a separate project.
>
>
> Thank you again for posting here on a non-commercial,
> mouseless-user-friendly list so I can respond :)
>
> Note: I never post benchmarks for my own servers:
> a) They'll inevitably come off as biased
> b) I don't want to make other servers look bad
>    (but I am always allowed to make my own servers look bad)
>
> Thus independent evaluations are very welcome :)
>
>
> [1] http://yahns.yhbt.net/README
>     a webserver by a guy who can't even do HTML :P)
>
> [2] I use C if performance is important, but usually Ruby is enough.
>