Issue #16352 has been updated by duerst (Martin D=FCrst).


jeremyevans0 (Jeremy Evans) wrote in #note-3:
> Supporting the dumping of larger objects does seem like a useful feature,=
 but as @shyouhei mentioned, it requires a format change, which would break=
 backwards compatibility as a marshal dump from Ruby 3 would not be restora=
ble on Ruby 2.7. It does seem like Ruby 3 would be a good time to implement=
 such a format change if we want to support the marshaling of larger object=
s.  We would probably want to keep the supporting the old format so that a =
marshal dump from Ruby 2.7 will work in Ruby 3, and maybe consider working =
on a gem that you could install in older Ruby versions to support the new m=
arshal format.

It sure would be useful to be able to dump huge stuff. But how frequent is =
this? My guess is that it would be better to by default use the current for=
mat, and switch to the new format with an explicit option. That might give =
the better interoperability story.


----------------------------------------
Feature #16352: Modify Marshal to dump objects larger than 2 GiB
https://bugs.ruby-lang.org/issues/16352#change-85962

* Author: seoanezonjic (Pedro Seoane)
* Status: Open
* Priority: Normal
----------------------------------------
Hi
Using a gem to handle matrix operations called Numo-array I found the follo=
wing error when save large matrix:
in `dump': long too big to dump (TypeError)
Github thread: https://github.com/ruby-numo/numo-narray/issues/144
Digging with the authors, we found the following code that reproduces the e=
rror:
```
ruby -e 'Marshal.dump(" "*2**31)'
```
Executed in :
ruby 2.7.0dev (2019-11-12T12:03:22Z master 3816622fbe) [x86_64-linux]

The marshal library  has a limit that is checked with the SIZEOF_LONG const=
ant. This check is performed in this line https://github.com/ruby/ruby/blob=
/e7ea6e078fecb70fbc91b04878b69f696749afac/marshal.c#L301 to 321 of the Mars=
hal.c file. I don't understand the motivation of this limit and has a great=
 impact in libraries that need to serialize large objects as numeric matrix=
. In this case, the limit of  >=3D 2 GiB it's reached easily and it blocks =
the ruby development in scientifical projects as cited. I found other bug r=
elated: #1560, but the Marshal problem itself was not addressed in this cas=
e.
Thank you in advance
PEdro Seoane



-- =

https://bugs.ruby-lang.org/

Unsubscribe: <mailto:ruby-core-request / ruby-lang.org?subject=3Dunsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-core>