Issue #15764 has been updated by duerst (Martin D=FCrst).


There may be a question as to what to do with encodings other than UTF-8. I=
 see three possibilities:

1) Ignore (i.e. leave as is), because who still uses them?

2) Use a rule based on character properties such as letter and digit that a=
re supported in these encodings

3) Convert to Unicode and check there

I would be okay with any one of these.

----------------------------------------
Bug #15764: Whitespace and control characters should not be permitted in to=
kens
https://bugs.ruby-lang.org/issues/15764#change-77647

* Author: BatmanAoD (Kyle Strand)
* Status: Open
* Priority: Normal
* Assignee: matz (Yukihiro Matsumoto)
* Target version: =

* ruby -v: =

* Backport: 2.4: UNKNOWN, 2.5: UNKNOWN, 2.6: UNKNOWN
----------------------------------------
As of Ruby 2.5.1p57, it appears that all valid Unicode code-points above 12=
8 are permitted in tokens. This includes whitespace and control characters.

This was demonstrated here: https://gist.github.com/qrohlf/7045823

I have attached the raw download from the above gist.

The issue has been discussed on StackOverflow: https://stackoverflow.com/q/=
34455427/1858225

I would say this is arguably a bug, but I am marking this ticket as a "feat=
ure" since the current behavior could be considered by-design.

---Files--------------------------------
helloworld.rb (543 Bytes)


-- =

https://bugs.ruby-lang.org/

Unsubscribe: <mailto:ruby-core-request / ruby-lang.org?subject=3Dunsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-core>