While debugging a Ruby 1.9 crash, I noticed some machine stack memory
leaks (unrelated to the crash).  Here is my analysis.

Compiler: gcc 3.4.6 on Linux (i686).


Leaky test program:

GC.start       # to help with debugging
@c = 0
@c += 1  while @c < 1000


Explanation:

Some INSNs in insns.def compile badly.  In general, INSN definitions
that have one or more function calls without any "if" conditions will
leak (usually about 16 bytes per call).  Some examples are
getinstancevariable, setinstancevariable, getspecial, setspecial,
getclassvarible, setclassvariable.

Here is an i386 disassembly of the getinstancevariable INSN, with my own
comments added:

mov    0x4(%edi),%eax
lea    0x8(%edi),%ebx
sub    $0x8,%esp                 # rb_ivar_get needs 8 bytes
mov    %ebx,(%esi)
push   %eax                      # + 4 bytes
pushl  0x14(%esi)                # + 4 bytes on the stack
call   0x80cd11c <rb_ivar_get>
mov    0x4(%esi),%edx
mov    %eax,(%edx)
add    $0x4,%edx
mov    %ebx,%edi
mov    %edx,0x4(%esi)
mov    (%ebx),%eax
jmp    *%eax             # this is inserted as an asm() statement,
                         #  so the compiler doesn't know it is jumping.
add    $0x10,%esp                # here the compiler cleans up the
                                 #  stack, but it's too late.
jmp    0x80d32e0 <vm_eval+36>


Possible solution:

I have noticed that a "goto *address" line causes gcc to emit a "jmp *%
eax" instruction anyway.  In that case, gcc knows it's a jump, so it
cleans up the stack before the jump.


GDB output to see the leak:

[root@vmware-cheath ruby19]# gdb /usr/local/bin/ruby19
GNU gdb Red Hat Linux (6.3.0.0-1.153.el4rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...Using host libthread_db library "/lib/tls/libthread_db.so.1".
 
(gdb) break garbage_collect
Breakpoint 1 at 0x8061210: file gc.c, line 1468.
(gdb) run teststack.rb
Starting program: /usr/local/bin/ruby19 teststack.rb
[Thread debugging using libthread_db enabled]
[New Thread -1208936768 (LWP 9943)]
[New Thread -1211765856 (LWP 9946)]
[Switching to Thread -1208936768 (LWP 9943)]
 
Breakpoint 1, garbage_collect () at gc.c:1468
1468        if (GC_NOTIFY) printf("start garbage_collect()\n");
(gdb) break rb_ivar_get
Breakpoint 2 at 0x80cd13f: file variable.c, line 968.
(gdb) display $sp
1: $sp = (void *) 0xbfed6784
(gdb) cont
Continuing.
 
Breakpoint 2, rb_ivar_get (obj=9489, id=151689584) at variable.c:968
968     {
1: $sp = (void *) 0xbfed7858
(gdb)
Continuing.
 
Breakpoint 2, rb_ivar_get (obj=9489, id=1) at variable.c:968
968     {
1: $sp = (void *) 0xbfed7848
(gdb)
Continuing.
 
Breakpoint 2, rb_ivar_get (obj=9489, id=3083392588) at variable.c:968
968     {
1: $sp = (void *) 0xbfed7828
(gdb)
Continuing.
 
Breakpoint 2, rb_ivar_get (obj=9489, id=1) at variable.c:968
968     {
1: $sp = (void *) 0xbfed7818
(gdb)
Continuing.
 
Breakpoint 2, rb_ivar_get (obj=9489, id=3083392588) at variable.c:968
968     {
1: $sp = (void *) 0xbfed77f8
(gdb)
Continuing.
 
Breakpoint 2, rb_ivar_get (obj=9489, id=1) at variable.c:968
968     {
1: $sp = (void *) 0xbfed77e8
(gdb)
Continuing.
 
Breakpoint 2, rb_ivar_get (obj=9489, id=3083392588) at variable.c:968
968     {
1: $sp = (void *) 0xbfed77c8
(gdb)
Continuing.
 
Breakpoint 2, rb_ivar_get (obj=9489, id=1) at variable.c:968
968     {
1: $sp = (void *) 0xbfed77b8
(gdb)
Continuing.
 
Breakpoint 2, rb_ivar_get (obj=9489, id=3083392588) at variable.c:968
968     {
1: $sp = (void *) 0xbfed7798
(gdb)
Continuing.
 
Breakpoint 2, rb_ivar_get (obj=9489, id=1) at variable.c:968
968     {
1: $sp = (void *) 0xbfed7788
(gdb)
Continuing.
 
Breakpoint 2, rb_ivar_get (obj=9489, id=3083392588) at variable.c:968
968     {
1: $sp = (void *) 0xbfed7768
(gdb)
Continuing.
 
Breakpoint 2, rb_ivar_get (obj=9489, id=1) at variable.c:968
968     {
1: $sp = (void *) 0xbfed7758
(gdb)
Continuing.
 
Breakpoint 2, rb_ivar_get (obj=9489, id=3083392588) at variable.c:968
968     {
1: $sp = (void *) 0xbfed7738

etc....