Issue #16352 has been updated by shevegen (Robert A. Heiler).


> I don't understand the motivation of this limit and has a great impact in libraries that need to serialize large objects as numeric matrix.
> In this case, the limit of >= 2 GiB it's reached easily and it blocks the ruby development in scientifical projects as cited.

Shyouhei already pointed out the historic reason. I believe you can quite easily convince the ruby core team that a change may
be necessary in the long run (most likely past ruby 3.0) based on use cases. Matz likes to hear real world use cases, so the
more information may be given the better. :)

As for possibility of change, I guess the Marshal format could be kept by default, but another variant could perhaps be added
where people could switch to another format - a bit like syck and psych could be used interchangably for yaml to some extent
(I used syck for quite some time even after psych was added, before I transitioned into Unicode finally; I used to specify
the yaml engine via e. g. YAML.engine = or something like that).

----------------------------------------
Bug #16352: Marshal limit of  >= 2 GiB
https://bugs.ruby-lang.org/issues/16352#change-82748

* Author: seoanezonjic (Pedro Seoane)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: 
* ruby -v: ruby 2.7.0dev (2019-11-12T12:03:22Z master 3816622fbe) [x86_64-linux]
* Backport: 2.5: UNKNOWN, 2.6: UNKNOWN
----------------------------------------
Hi
Using a gem to handle matrix operations called Numo-array I found the following error when save large matrix:
in `dump': long too big to dump (TypeError)
Github thread: https://github.com/ruby-numo/numo-narray/issues/144
Digging with the authors, we found the following code that reproduces the error:
```
ruby -e 'Marshal.dump(" "*2**31)'
```
Executed in :
ruby 2.7.0dev (2019-11-12T12:03:22Z master 3816622fbe) [x86_64-linux]

The marshal library  has a limit that is checked with the SIZEOF_LONG constant. This check is performed in this line https://github.com/ruby/ruby/blob/e7ea6e078fecb70fbc91b04878b69f696749afac/marshal.c#L301 to 321 of the Marshal.c file. I don't understand the motivation of this limit and has a great impact in libraries that need to serialize large objects as numeric matrix. In this case, the limit of  >= 2 GiB it's reached easily and it blocks the ruby development in scientifical projects as cited. I found other bug related: #1560, but the Marshal problem itself was not addressed in this case.
Thank you in advance
PEdro Seoane



-- 
https://bugs.ruby-lang.org/

Unsubscribe: <mailto:ruby-core-request / ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-core>