ko1 / atdot.net wrote:
> sorry for long absent about this topic. it is hard task (hard
> to start writing up because of problem difficulties and my
> English skil ;p ) to summarize about this topic.

No problem, thank you for summarizing.

> I try to write step by step.
> 
> ----
> 
> # Discussion at last developers meeting
> 
> ## Thread/Fiber switch safety
> 
> Koichi: (repeat my opinion about difficulty of thread/fiber safety)
> 
> akr: providing better synchronize mechanism (such as go-lang
> has) and encouraging safe parallel computation seems better.
> 
> Koichi: It is one possible solution but my position is "if
> people can shoot their foot, people will shoot".

I think your approach is too cautious.

We already have many dangerous things in Ruby, even in
single-threaded code.  For example: File.read, IO#read, IO#gets
are all dangerous with no size limit: they can cause
out-of-memory or swapping on gigantic inputs, leading to DoS.

Fork and inadvertant sharing of open files/sockets can also
cause problems.  And there are also pathological Regexp which
can cause unbound CPU usage.

> Matz: I don't like to force people to use lock and so on.
> 
> (the point is Matz doesn't reject "-safe" approach)
> 
> ## Introduce restriction
> 
> (The following idea is not available at last meeting (only
> part of idea I showed))
> 
> Koichi:
> 
> The problem of this feature is mind gap using auto-fiber user
> and script writer. This is same as thread-safety. Person A
> consider the code is auto-fiber safe, and other person B (can
> be same as A) write a code without auto-fiber safety, then it
> will be problem.
> 
> In general, most of existing libraries are not auto-fiber safe
> code (it doesn't mean most of libraries are not auto-fiber
> safe. Many code are auto-fiber safe without any care).

Right; most code does not have to care; and all these dangers
already exist with native Threads.

> If we can know a code (and code called by this code) is
> auto-fiber safe, we can use auto-fiber in safe.
>  
> There are three type of code.
> 
> * (1) don't care about auto-fiber
> * (2) auto-fiber aware code (assume switching is not allowed at the beginning)
> * (3) auto-fiber aware code (don't care it is allowed or not allowed to switch)
> 
> There are three types of status.
> 
> * (a) can't switch
> * (b) can enable to switch, but don't switch
> * (c) can switch
> 
> in matrix
> 
> ```
>     can switch / can enable switch
> (a) can't      / can't
> (b) can't      / can
> (c) can        / ??
> ```
> 
> matrix with (1-3) and (a-c)
> 
> ```
>      (a)     (b)     (c)
> (1)   OK      NG      NG
> (2)   OK      OK      NG
> (3)   OK(*1)  OK(*1)  OK
> ```
> 
> (1)-(b) and (1)-(c) is not accepted because other method called from this code can switch the context.
> (2)-(c) is also unacceptable because the beginning of code is not auto-fiber aware.
> 
> *1) Possible problem: (3) can introduce dead-lock problem because it can stop forever.

Perhaps holding Mutex lock should disable auto-fiber switching.
This should prevent deadlocks, I think.

Existing code has Mutexes, so I'm not sure how they should
interact with auto-Fiber.  I agree with Matz that we should
discourage locking, so I guess disabling auto-Fiber switch
while Mutex is held is the most straightforward solution.

> Normal threads start from (a).  Auto-fibers start from (b).
> They are written in (1), (2) and (3). Maybe (2) is written for
> auto fiber top-lelvel. This code will call some async methods
> which can change context.
> 
> My proposal is, to write down explicitly of (1) to (3) and (a)
> to (c) in program.
> 
> At the meeting, I proposed non-matured keywords(-like) to control them.
> (and just now I don't have good syntax for it yet)
> 
> akira: If we introduce such keywords, we need to rewrite all
> of code if we want to use auto-fiber web application request
> handler (for example, we need to rewrite Rails to run on
> auto-fiber based rack server).
> 
> Matz: it is unacceptable to introduce huge rewriting for existing code.

I agree completely with akira's observation and Matz's opinion
of this.

> (IMO (not appeared in last meeting) we need to rewrite all of
> code even if we don't introduce keywords to make sure the
> auto-fiber safety) 

I don't agree with this.  A lot of code is already auto-fiber
safe because they are written with GVL+Threads in mind.
(see my original Net::HTTP example); and we also have a lot
of code (webrick, net/*) which worked fine with green Threads
in 1.8

Worst case is we release GVL in a native Thread and forget to
yield to other Fibers in the same Thread.  However, that is
already a problem with existing code when run inside Fibers
(e.g.  getaddrinfo, IO operations on NFS/slow-disk, ...)

I am working on making rb_thread_fd_select auto-fiber aware,
too.  (done for iom_select/iom_epoll, working on iom_kqueue)

> Matz and I discussed about this issue, and we conclude that it
> is too early to introduce this feature on Ruby 2.5.

OK, I will continue to work on implementation improvements
and keep patches rebased to trunk.

> I want to consider this issue further. auto-fiber based guild
> is one possibility, this mean we can introduce object
> isolation and context switching each other.

Do you think this is in the 2.5 timeline?

Thank you.

Unsubscribe: <mailto:ruby-core-request / ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-core>