samuel / oriontransfer.org wrote:
> Thanks for your detailed reply. It's impressive and useful
> that you have such a good knowledge of these issues.

No problem.

> I spent some time just thinking about this issue, and how this
> feature tries to solve the problem in Ruby.
> 
> On the one hand, I'm fundamentally opposed to increasing the
> surface area of Ruby when it could be done by writing a gem.
> This has a massive upstream cost, affecting both JRuby and
> Rubinius. While I appreciate what you are saying w.r.t.
> maximising usage, I feel like building this into Ruby will
> cause stagnation of progress long term - one solution for all
> problems isn't always ideal. Seeing initiatives like
> stdgems.org only reinforces how I feel about this.

I understood something it was already decided by matz and ko1 to
do something along the lines of auto-Fiber.  Though I can't find
ko1's original message in the archives, it's mostly quoted in in
my reply to him:

  http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/80531

I should note some languages like Go, Erlang, Haskell, and the
afore-mentioned Crystal all have lightweight threading along
these lines in the core language.

In their current state, Fibers are much less useful than the
equivalents in those languages; while native Threads are too
expensive.  Something in between Fibers and Threads seems
desirable; maybe we can give auto-Fiber another (short) name;
but I'm not sure it's necessary.

I was also influenced to explore lightweight threading in a
rack-devel thread and the responses James Tucker wrote to me:

  Subject: big responses to slow clients: Rack vs PSGI

It's somewhere in https://groups.google.com/group/rack-devel but
that requires JS; so I can't view or link to it using w3m :<

> Generally speaking - I really appreciate the work that's been
> done here. I also feel like you've reinvented nio4r, async and
> a bunch of other stuff, at a very low level, without as much
> testing, compatibility, etc.

That's a fair point about less testing and compatibility.
But, I think there is more code using normal Ruby stdlib
that can automatically take advantage of these changes
so we'll be able to nail down any problems quickly.

On a technical level, I consider the design of libev (used by
nio4r and async) too limited in that it does not take advantage
of thread-safety baked into kqueue and epoll.  Thinking in terms
of "events vs. threads" too limiting.  As I've said before;
combining them is advantageous because both have their uses.
kqueue is an thread-friendly queue, so is epoll.


This feels like the microkernel vs monolithic kernel debate,
too.  On one level, isolation and compartmentalization provided
by micro-kernels is appealing; but the ease-of-development of a
monolith allowed Linux to become the kernel for nearly
everything, from tiny IoT devices to giant supercomputers.

And that doesn't preclude things like loadable modules and FUSE
for userspace filesystems from being useful, despite core
filesystem drivers being bundled with Linux.  So I think `async`
can still be supported as an alternative for Ruby; but the
bundled implementation can benefit more from tighter integration
into the core.


A more recent example might be git; which included high-level
non-essential "porcelain" tools early on in addition to the core
"plumbing".  Initially, it was intended that separately
maintained wrappers such as "cogito", would implement the
porcelain UI bits and git would remain low-level plumbing.  That
ended up making both development and usage more complicated.
Eventually git swallowed up most of the cogito functionality and
cogito was abandoned.

git also ended up with bundled functionality that would've been
separately packaged in other VCSes, including import/export
tools for email, CVS, SVN, etc.

The most relevant example from git might be the bundling of
libxdiff in git, allowing optimizations and tweaks not possible
with an external diff.  However, GIT_EXTERNAL_DIFF still remains
supported for less-common use cases.


On a non-technical level:

Finally, this (ruby-core) is one of the few places I can still
contribute to in the Ruby world.  All other relevant Ruby
projects requires running non-Free software (including JS) and
having to abide accept Terms-of-Service set by a corporation.

Fwiw, I agree with Rubinius philosophy of implementing more of
Ruby in Ruby and would rather contribute to that; but the above
is a huge factor in why I went on to work on C Ruby, instead.
(the other major factor is I strongly prefer C to C++).

> Ideally, we could move all socket related code into a gem -
> perhaps that's already on the cards e.g. stdgems. Once that's
> done, fixing issues like `exceptions: false` would be easier
> since it can be versioned.

Maybe that'll be done, too, but not my call.
But what about IO.pipe, `backtick`, and IO.popen?

> I was thinking about how we could expose this to Ruby - and
> ideally, I think we should add two functions:
> 
> `IO.wait_for_single_fd` and `IO.wait_for_pid`. The C functions
> rb_wait_for_single_fd and rb_waitpid would invoke these
> functions, and these functions would implement the current
> logic of the current C functions. It probably makes sense to
> think in more detail how these functions should work - e.g.
> `wait_for_multiple_fds` (or `select`), or something more
> elaborate.

Maybe.. I guess we already have IO#wait_*able in io/wait; and
Process.wait*/IO.select is already possible to override and that
would have the same effect.  We'd also have to expose the
optional read/write buffering + encoding conversion and make
that accessible to pure Ruby.

It would make C Ruby feel closer to Rubinius and that would be
nice :)  I'm not sure how feasable it would be; to introduce
more Ruby-visible APIs to implement this.

And I think exposing more APIs to handle FDs directly is a
mistake in the presence of native threads.  My proposed C API
prefers "int *fd" and "rb_io_t" to deal with close notification
handling.  Multithreaded programs recycle FDs frequently and
internal APIs need to be prepared to deal with that.

The implementation I proposed also takes advantage of some
C-only optimizations such as reading/writing to memory across
Fiber stack boundaries: something which cannot be done with
higher-level APIs .  Similar optimizations already landed for
thread_sync.c (Mutex/Queue) as well as IO#close in trunk.

Again, designing user-visible APIs is most difficult and
ruby-core have to think most about long-term support and
consequences.

So the difficulty of changing/adding APIs is:

	1) internal C API (easiest)
	2) public C API (difficult)
	3) Ruby API (most difficult)

So, I've mainly done 1) and made minimal additions to 3).
Only changes to 2) are to internal behavior, so use from C
extensions remains the same.

> Then, we could allow things like `async` and `auto-fibers` to
> extend Ruby's IO system to provide a policy for blocking IO.
> `auto-fibers` could be implemented as a gem with a C
> extension.
> 
> What do you think?

I guess this is meant for matz and ko1.

We could actually have that today; and I guess you already have
that with `async`.  All the IO methods are well-documented and
you can even ignore/override the existing IO buffering if you
override all the methods by monkey patching core classes.
Heck, you may even go as far as to never allocate rb_io_t
if you override IO.open/IO.pipe/*Socket.new/... and replace
them with your own class.

What I think is (or at least ought to be) irrelevant.

I only give matz and ko1 another option to choose from.  We can wait
for matz and ko1 to decide what to do, maybe they'll discuss
this at: https://bugs.ruby-lang.org/projects/ruby/wiki/DevelopersMeeting20170616Japan
I certainly won't attend meetings or try to influence anybody
using anything besides plain-text messages, here.

Unsubscribe: <mailto:ruby-core-request / ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-core>