This is a multi-part message in MIME format.
--------------030706030406040700000602
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Help!

Our Ruby controlled Robotic Marine Laboratory started failing
with segmentation faults just a few days before it was to be
deployed.  We had seen random, very occasional segmentation
faults for some months, as our application grew larger
and more complex.  Then, just days before the ship
was scheduled to sail, after we'd integrated a couple new and
exciting features, we started getting segfaults regularly.

We have not been able to correlate the fault to a particular operation
in our application.  However, it never occurs sooner than 30 minutes
or so after the application starts.  The application is heavily
multithreaded and uses 3 RS-232 serial ports accessed via
the ruby termios library.

Can anyone give me any hints as to what's going on by
looking at the attached stack backtrace?

It appears to be deep into a GC cycle while executing Marshal.dump.

Some notes on the target environment:
    ARM9 processor (ep9312) running Linux kernel v2.4.26
    (cross-)Compiled with gcc version 3.4.5  -Os  (not omitting
framepointer!)
    glibc 2.2.5
    Ruby 1.6.8
    64MB RAM
    100MB swap (on a compact flash card)

Here's the CPU board:
    http://www.embeddedarm.com/epc/ts7200-spec-h.html

Notes looking at the backtrace:
    The only "unusual" 'C' shared library is termios
    The interpreter was running GC at the time of failure
    There is also an instance of Marshal.dump on the stack
       (we use this to serialize objects for our own execution logs)

We've managed to band-aid around the problem by soldering 32MB
more RAM onto the board and running the
outer level logic as a shell script that invokes ruby repeatedly to
run the guts of the app.  I disable the garbage collector just
as late as I can without the Ruby process consuming all available
RAM and swap before the application can exit normally
back into the outer shell script.

It's been limping along this way for the last 5 days
without crashing.  I'm really hoping that someone on this list can
at least suggest to me some debugging techniques for
(what seems to be) a GC related problem.

-- 
 Brent Roman                                   MBARI
 Software Engineer               Tel: (831) 775-1808
 7700 Sandholdt Road,         Moss Landing, CA 95039
 mailto:brent / mbari.org  http://www.mbari.org/~brent


--------------030706030406040700000602
Content-Type: text/plain;
 namet.07may14larv"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
 filenamet.07may14larv"

Setting up the environment for debugging ARM target in gdb.
The target architecture is assumed to be ep9312
During symbol reading, incomplete CFI data; unspecified registers (e.g., r0) at 0x283ac.
Core was generated by `/home/esp/esp2/mission/07may14.rb                               '.
Program terminated with signal 11, Segmentation fault.
#0  mark_locations_array (x0x1, nq8165567) at gc.c:368
368	{
#0  mark_locations_array (x1, nq8165567) at gc.c:368
#1  0x000285ac in rb_gc_mark_frame (frame433184) at gc.c:1165
#2  0x00016134 in thread_mark (th28dfe8) at eval.c:7422
#3  0x0002823c in rb_gc_mark (ptr at gc.c:878
#4  0x0002844c in mark_locations_array (x66977c, n5932) at gc.c:434
#5  0x00016040 in thread_mark (th407c68) at eval.c:7399
#6  0x0002823c in rb_gc_mark (ptr at gc.c:878
#7  0x00028570 in mark_entry (key valueq8165568) at gc.c:461
During symbol reading, unsupported const value attribute form: 'DW_FORM_strp'.
#8  0x0005c9f0 in st_foreach (table68c8f8, func28564 <mark_entry>, arg0) at st.c:488
#9  0x00028200 in rb_gc_mark (ptr at gc.c:861
#10 0x00028570 in mark_entry (key valueq8165568) at gc.c:461
#11 0x0005c9f0 in st_foreach (table664f98, func28564 <mark_entry>, arg0) at st.c:488
#12 0x00067864 in rb_mark_generic_ivar (objq8165568) at variable.c:912
#13 0x00027d80 in rb_gc_mark (ptr at gc.c:697
#14 0x00028570 in mark_entry (key valueq8165568) at gc.c:461
#15 0x0005c9f0 in st_foreach (tablef1480, func28564 <mark_entry>, arg0) at st.c:488
#16 0x00028570 in mark_entry (key valueq8165568) at gc.c:461
#17 0x0005c9f0 in st_foreach (table193410, func28564 <mark_entry>, arg0) at st.c:488
#18 0x0002855c in mark_hashentry (key valueq9186048) at gc.c:479
#19 0x0005c9f0 in st_foreach (table162ba0, func28548 <mark_hashentry>, arg0)
    at st.c:488
#20 0x00028214 in rb_gc_mark (ptr at gc.c:866
#21 0x00028570 in mark_entry (key valueq8165568) at gc.c:461
#22 0x0005c9f0 in st_foreach (tabled68f8, func28564 <mark_entry>, arg0) at st.c:488
#23 0x000281d4 in rb_gc_mark (ptr at gc.c:850
#24 0x00028138 in rb_gc_mark (ptr at gc.c:845
#25 0x00028200 in rb_gc_mark (ptr at gc.c:861
#26 0x00028570 in mark_entry (key valueq8165568) at gc.c:461
#27 0x0005c9f0 in st_foreach (table115b80, func28564 <mark_entry>, arg0) at st.c:488
#28 0x00028264 in rb_gc_mark (ptr at gc.c:900
#29 0x00028570 in mark_entry (key valueq8165568) at gc.c:461
#30 0x0005c9f0 in st_foreach (table16b8f8, func28564 <mark_entry>, arg0) at st.c:488
#31 0x000281dc in rb_gc_mark (ptr at gc.c:851
#32 0x000281d4 in rb_gc_mark (ptr at gc.c:850
#33 0x000281d4 in rb_gc_mark (ptr at gc.c:850
#34 0x000281d4 in rb_gc_mark (ptr at gc.c:850
#35 0x00028138 in rb_gc_mark (ptr at gc.c:845
#36 0x00028570 in mark_entry (key valueq8165568) at gc.c:461
#37 0x0005c9f0 in st_foreach (table97750, func28564 <mark_entry>, arg0) at st.c:488
#38 0x00028570 in mark_entry (key valueq8165568) at gc.c:461
#39 0x0005c9f0 in st_foreach (table97a10, func28564 <mark_entry>, arg0) at st.c:488
#40 0x000281d4 in rb_gc_mark (ptr at gc.c:850
#41 0x000281d4 in rb_gc_mark (ptr at gc.c:850
#42 0x00028138 in rb_gc_mark (ptr at gc.c:845
#43 0x00028138 in rb_gc_mark (ptr at gc.c:845
#44 0x00028138 in rb_gc_mark (ptr at gc.c:845
#45 0x0002844c in mark_locations_array (x7ffe6c54, n at gc.c:434
#46 0x000285ac in rb_gc_mark_frame (frame7ffe6bb8) at gc.c:1165
#47 0x00028d40 in rb_gc () at gc.c:1241
#48 0x00029460 in ruby_xmalloc (size) at gc.c:109
#49 0x0005c6bc in st_add_direct (table1b1ef8, key36d5 <Address 0x36d5 out of bounds>, 
    valuee <Address 0xe out of bounds>) at st.c:313
#50 0x00032230 in w_symbol (id037, arg7ffe6a0c) at marshal.c:206
#51 0x0003290c in w_object (objq9164948, arg7ffe6a0c, limit) at marshal.c:462
#52 0x00032924 in w_object (objq9164088, arg7ffe6a0c, limit) at marshal.c:463
#53 0x00032b0c in dump (arg94fe8) at marshal.c:502
#54 0x00012464 in rb_ensure (b_proc32af4 <dump>, data1!47379712, 
    e_proc32b14 <dump_ensure>, data2!47379724) at eval.c:4070
#55 0x00032cbc in marshal_dump (argc!47379724, argv2ace5640) at marshal.c:567
#56 0x00012a38 in call_cfunc (func32b68 <marshal_dump>, recvq8130348, len!47379076, 
    argc!47339264, argv7ffe6c54) at eval.c:4285
#57 0x0001aac8 in rb_call0 (klassq8130288, recvq8130348, id@65, argc argv7ffe6c54, 
    body2acdcc5c, flags at eval.c:4423
#58 0x0001b3d0 in rb_call (klassq8130288, recvq8130348, mid@65, argc argv7ffe6c54, 
    scope at eval.c:4654
#59 0x00018a78 in rb_eval (selfq9164088, n2ace5640) at ruby.h:574
#60 0x00018708 in rb_eval (selfq9164088, n2ace5640) at eval.c:2556
#61 0x0001af54 in rb_call0 (klassq8187808, recvq9164088, id9, argc argv0, 
    body2acd3be8, flags at eval.c:4560
#62 0x0001b3d0 in rb_call (klassq8187808, recvq9164088, mid9, argc argv0, scope
    at eval.c:4654
#63 0x00018a78 in rb_eval (selfq9163048, n2ace5640) at ruby.h:574
#64 0x000193b4 in rb_eval (selfq9163048, n2ace5640) at eval.c:2729
#65 0x00018694 in rb_eval (selfq9163048, n2ace5640) at eval.c:2555
#66 0x0001728c in rb_eval (selfq9163048, n2ace5640) at eval.c:2058
#67 0x0001af54 in rb_call0 (klassq8677848, recvq9163048, id929, argc argv7ffe8ec4, 
    body2ad68784, flags at eval.c:4560
#68 0x0001b3d0 in rb_call (klassq8677848, recvq9163048, mid929, argc  argv7ffe8ebc, 
    scope at eval.c:4654
#69 0x00018a78 in rb_eval (selfq9163048, n2ace5640) at ruby.h:574
#70 0x0001af54 in rb_call0 (klassq8677848, recvq9163048, id673, argc argv0, 
    body2ad69d50, flags at eval.c:4560
#71 0x0001b3d0 in rb_call (klassq8677848, recvq9163048, mid673, argc argv0, 
    scope  at eval.c:4654
#72 0x00018a78 in rb_eval (selfq9163048, n2ace5640) at ruby.h:574
#73 0x0001728c in rb_eval (selfq9163048, n2ace5640) at eval.c:2058
#74 0x0001af54 in rb_call0 (klassq8677848, recvq9163048, idp09, argc argv7ffea1bc, 
    body2ad66fb0, flags at eval.c:4560
#75 0x0001b3d0 in rb_call (klassq8677848, recvq9163048, midp09, argc  argv7ffea1b4, 
    scope at eval.c:4654
#76 0x00018a78 in rb_eval (selfq9163048, n2ace5640) at ruby.h:574
#77 0x000181f0 in rb_eval (selfq9163048, n2ace5640) at eval.c:2441
#78 0x0001af54 in rb_call0 (klassq8677848, recvq9163048, id&353, argc argv7ffeadd8, 
    body2ad656d8, flags at eval.c:4560
#79 0x0001b3d0 in rb_call (klassq8677848, recvq9163048, mid%873, argc  argv7ffeadd0, 
    scope at eval.c:4654
#80 0x00018a78 in rb_eval (selfq9448868, n2ace5640) at ruby.h:574
#81 0x0001728c in rb_eval (selfq9448868, n2ace5640) at eval.c:2058
#82 0x0001af54 in rb_call0 (klassq9444528, recvq9448868, id(185, argc argv7ffeb9e4, 
    body2ada696c, flags at eval.c:4560
#83 0x0001b3d0 in rb_call (klassq9444528, recvq9448868, mid(185, argc argv7ffeb9e0, 
    scope at eval.c:4654
#84 0x00018a78 in rb_eval (selfq9448868, n2ace5640) at ruby.h:574
#85 0x0001728c in rb_eval (selfq9448868, n2ace5640) at eval.c:2058
#86 0x0001af54 in rb_call0 (klassq9444528, recvq9448868, id(345, argc argv7ffec610, 
    body2ada34b0, flags at eval.c:4560
#87 0x0001b3d0 in rb_call (klassq9444528, recvq9448868, mid(345, argc0 argv7ffec604, 
    scope at eval.c:4654
#88 0x00018a78 in rb_eval (selfq9219008, n2ace5640) at ruby.h:574
#89 0x0001af54 in rb_call0 (klassq9220568, recvq9219008, id(633, argc argv0, 
    body2ad7d4e0, flags at eval.c:4560
#90 0x0001b3d0 in rb_call (klassq9220568, recvq9219008, mid(633, argc argv0, 
    scope at eval.c:4654
#91 0x00018a78 in rb_eval (selfq8183648, n2ace5640) at ruby.h:574
#92 0x000181f0 in rb_eval (selfq8183648, n2ace5640) at eval.c:2441
#93 0x0001af54 in rb_call0 (klassq8187808, recvq8183648, id)337, argc argv7ffed924, 
    body2ad05074, flags  at eval.c:4560
#94 0x0001b3d0 in rb_call (klassq8187808, recvq8183648, mid)337, argc  argv7ffed91c, 
    scope at eval.c:4654
#95 0x00018a78 in rb_eval (selfq8183648, n2ace5640) at ruby.h:574
#96 0x00017bd8 in rb_eval (selfq8183648, n2ace5640) at eval.c:2309
#97 0x0001cf0c in rb_yield_0 (valr0749732, selfq8183648, klass acheck at eval.c:3650
#98 0x00017f94 in rb_eval (selfq8183648, n2ace5640) at eval.c:2391
#99 0x0001728c in rb_eval (selfq8183648, n2ace5640) at eval.c:2058
#100 0x0001af54 in rb_call0 (klassq8187808, recvq8183648, id0953, argc 
    argv7ffef0fc, body2ad7e32c, flags  at eval.c:4560
#101 0x0001b3d0 in rb_call (klassq8187808, recvq8183648, mid0953, argc0 
    argv7ffef0f0, scope at eval.c:4654
#102 0x00018a78 in rb_eval (selfq8183648, n2ace5640) at ruby.h:574
#103 0x0001e910 in block_pass (selfq8183648, node2ad91328) at eval.c:6596
#104 0x00017a64 in rb_eval (selfq8183648, n2ace5640) at eval.c:2295
#105 0x0001af54 in rb_call0 (klassq8187808, recvq8183648, id0025, argc  
    argv7ffefe7c, body2ad913f0, flags  at eval.c:4560
#106 0x0001b3d0 in rb_call (klassq8187808, recvq8183648, mid0025, argc  
    argv7ffefe7c, scope at eval.c:4654
#107 0x00018a78 in rb_eval (selfq8183648, n2ace5640) at ruby.h:574
#108 0x00017bd8 in rb_eval (selfq8183648, n2ace5640) at eval.c:2309
#109 0x0001728c in rb_eval (selfq8183648, n2ace5640) at eval.c:2058
#110 0x0001af54 in rb_call0 (klassq8187808, recvq8183648, id1097, argc 
    argv7fff0fd0, body2afbb5a8, flags  at eval.c:4560
#111 0x0001b3d0 in rb_call (klassq8187808, recvq8183648, mid1097, argc  
    argv7fff0fc8, scope at eval.c:4654
#112 0x00018a78 in rb_eval (selfq8183648, n2ace5640) at ruby.h:574
#113 0x0001e72c in block_pass (selfq8183648, node2afbae14) at eval.c:6562
#114 0x00017a64 in rb_eval (selfq8183648, n2ace5640) at eval.c:2295
#115 0x0001af54 in rb_call0 (klassq8187808, recvq8183648, id1125, argc argv0, 
    body2afbaedc, flags  at eval.c:4560
#116 0x0001b3d0 in rb_call (klassq8187808, recvq8183648, mid1125, argc argv0, 
    scope at eval.c:4654
#117 0x00018a78 in rb_eval (selfq8183648, n2ace5640) at ruby.h:574
#118 0x0001728c in rb_eval (selfq8183648, n2ace5640) at eval.c:2058
#119 0x00021588 in rb_load (fnameq8404756, wrapa7468) at eval.c:5284
#120 0x000217fc in rb_f_load (argc argv2ace5640) at eval.c:5333
#121 0x00012a38 in call_cfunc (func217c8 <rb_f_load>, recvq8187688, len!47379076, 
    argc!47339264, argv7fff2b1c) at eval.c:4285
#122 0x0001aac8 in rb_call0 (klassq8186288, recvq8187688, id9, argc argv7fff2b1c, 
    body2acddc24, flags at eval.c:4423
#123 0x0001b3d0 in rb_call (klassq8186288, recvq8187688, mid9, argc argv7fff2b1c, 
    scope at eval.c:4654
#124 0x00018a78 in rb_eval (selfq8445576, n2ace5640) at ruby.h:574
#125 0x0001af54 in rb_call0 (klassq8444076, recvq8445576, id7, argc argv7fff3200, 
    body2acd7fcc, flags at eval.c:4560
#126 0x0001b3d0 in rb_call (klassq8444076, recvq8445576, mid7, argc argv7fff31fc, 
    scope at eval.c:4654
#127 0x00018a78 in rb_eval (selfq8445576, n2ace5640) at ruby.h:574
#128 0x00018574 in rb_eval (selfq8445576, n2ace5640) at eval.c:2529
#129 0x0001728c in rb_eval (selfq8445576, n2ace5640) at eval.c:2058
#130 0x0001af54 in rb_call0 (klassq8444076, recvq8445576, id9, argc argv7fff4344, 
    body2acd7c70, flags at eval.c:4560
#131 0x0001b3d0 in rb_call (klassq8444076, recvq8445576, mid9, argc  argv7fff433c, 
    scope at eval.c:4654
#132 0x00018a78 in rb_eval (selfq8445576, n2ace5640) at ruby.h:574
#133 0x0001af54 in rb_call0 (klassq8187808, recvq8445576, id5, argc argv7fff4a28, 
    body2acd5754, flags  at eval.c:4560
#134 0x0001b3d0 in rb_call (klassq8187808, recvq8445576, mid5, argc argv7fff4a24, 
    scope at eval.c:4654
#135 0x00018a78 in rb_eval (selfq8445576, n2ace5640) at ruby.h:574
#136 0x0001af54 in rb_call0 (klassq8444076, recvq8445576, id7, argc argv0, 
    body2acdbde8, flags at eval.c:4560
#137 0x0001b3d0 in rb_call (klassq8444076, recvq8445576, mid7, argc argv0, 
    scope  at eval.c:4654
#138 0x00018a78 in rb_eval (selfq8445576, n2ace5640) at ruby.h:574
#139 0x0001cf0c in rb_yield_0 (valq8938928, selfq8445576, klass acheck at eval.c:3650
#140 0x00017f94 in rb_eval (selfq8938968, n2ace5640) at eval.c:2391
#141 0x0001cf0c in rb_yield_0 (val@ selfq8938968, klass acheck at eval.c:3650
#142 0x00017f94 in rb_eval (selfq8165568, n2ace5640) at eval.c:2391
#143 0x00018014 in rb_eval (selfq8165568, n2ace5640) at eval.c:2401
#144 0x000181f0 in rb_eval (selfq8165568, n2ace5640) at eval.c:2441
#145 0x0001af54 in rb_call0 (klassq8166968, recvq8165568, id049, argc 
    argv7fff6f4c, body2ad4c62c, flags at eval.c:4560
#146 0x0001b3d0 in rb_call (klassq8166968, recvq8165568, mid049, argc 
    argv7fff6f48, scope at eval.c:4654
#147 0x00018a78 in rb_eval (selfq8938968, n2ace5640) at ruby.h:574
#148 0x00017bd8 in rb_eval (selfq8938968, n2ace5640) at eval.c:2309
#149 0x000181f0 in rb_eval (selfq8938968, n2ace5640) at eval.c:2441
#150 0x0001cf0c in rb_yield_0 (valq8165568, selfq8938968, klass acheck  at eval.c:3650
#151 0x00022ed8 in rb_thread_start_0 (fn2315c <rb_thread_yield>, arg2ada2308, 
    th_arg7ffdcc00) at eval.c:8592
#152 0x00012a1c in call_cfunc (func231f4 <rb_thread_initialize>, recvq8938968, 
    len!47379076, argc!47339264, argv7fff83a8) at eval.c:4282
#153 0x0001aac8 in rb_call0 (klassq8166968, recvq8938968, id(65, argc argv7fff83a8, 
    body2ace5b68, flags0 at eval.c:4423
#154 0x0001b3d0 in rb_call (klassq8166968, recvq8938968, mid(65, argc argv7fff83a8, 
    scope0 at eval.c:4654
#155 0x00018d54 in rb_eval (selfq8938968, n2ace5640) at eval.c:2606
#156 0x00017bd8 in rb_eval (selfq8938968, n2ace5640) at eval.c:2309
#157 0x0001af54 in rb_call0 (klassq8455656, recvq8938968, id(65, argc argv7fff9224, 
    body2acbfa08, flags  at eval.c:4560
#158 0x0001b3d0 in rb_call (klassq8455656, recvq8938968, mid(65, argc  argv7fff921c, 
    scope at eval.c:4654
#159 0x0001b688 in rb_funcall2 (recv!47379076, mid2264, argc  argv7fff921c)
    at ruby.h:574
#160 0x0001b814 in rb_obj_call_init (objq8938968, argc  argv7fff921c) at eval.c:5737
#161 0x000231c0 in rb_thread_s_new (argc  argv7fff921c, klassq8455656) at eval.c:8673
#162 0x00012a38 in call_cfunc (func23194 <rb_thread_s_new>, recvq8455656, len!47379076, 
    argc!47339264, argv7fff921c) at eval.c:4285
#163 0x0001aac8 in rb_call0 (klassq8166948, recvq8455656, id177, argc  argv7fff921c, 
    body2ace5b90, flags at eval.c:4423
#164 0x0001b3d0 in rb_call (klassq8166948, recvq8455656, mid177, argc  argv7fff921c, 
    scope at eval.c:4654
#165 0x00018a78 in rb_eval (selfq8455656, n2ace5640) at ruby.h:574
#166 0x0001e910 in block_pass (selfq8455656, node2acbf080) at eval.c:6596
#167 0x00017a64 in rb_eval (selfq8455656, n2ace5640) at eval.c:2295
#168 0x0001af54 in rb_call0 (klassq8453616, recvq8455656, id081, argc 
    argv7fff9fb0, body2acbf148, flags at eval.c:4560
#169 0x0001b3d0 in rb_call (klassq8453616, recvq8455656, mid081, argc 
    argv7fff9fac, scope at eval.c:4654
#170 0x00018a78 in rb_eval (selfq8445576, n2ace5640) at ruby.h:574
#171 0x0001e910 in block_pass (selfq8445576, node2acbd348) at eval.c:6596
#172 0x00017a64 in rb_eval (selfq8445576, n2ace5640) at eval.c:2295
#173 0x0001af54 in rb_call0 (klassq8187668, recvq8445576, idV45, argc argv7fffad30, 
    body2acbd410, flags at eval.c:4560
#174 0x0001b3d0 in rb_call (klassq8187668, recvq8445576, midV45, argc argv7fffad30, 
    scope at eval.c:4654
#175 0x00018a78 in rb_eval (selfq8445576, n2ace5640) at ruby.h:574
#176 0x00017bd8 in rb_eval (selfq8445576, n2ace5640) at eval.c:2309
#177 0x00019440 in rb_eval (selfq8445576, n2ace5640) at eval.c:2749
#178 0x0001728c in rb_eval (selfq8445576, n2ace5640) at eval.c:2058
#179 0x0001af54 in rb_call0 (klassq8444076, recvq8445576, id9, argc argv7fffc3a8, 
    body2ad16dec, flags at eval.c:4560
#180 0x0001b3d0 in rb_call (klassq8444076, recvq8445576, mid9, argc argv7fffc3a4, 
    scope at eval.c:4654
#181 0x00018a78 in rb_eval (selfq8445576, n2ace5640) at ruby.h:574
#182 0x000181f0 in rb_eval (selfq8445576, n2ace5640) at eval.c:2441
#183 0x00018014 in rb_eval (selfq8445576, n2ace5640) at eval.c:2401
#184 0x000181f0 in rb_eval (selfq8445576, n2ace5640) at eval.c:2441
#185 0x0001af54 in rb_call0 (klassq8444076, recvq8445576, id9, argc argv7fffda28, 
    body2acd59ac, flags at eval.c:4560
#186 0x0001b3d0 in rb_call (klassq8444076, recvq8445576, mid9, argc argv7fffda24, 
    scope at eval.c:4654
#187 0x00018a78 in rb_eval (selfq8445576, n2ace5640) at ruby.h:574
#188 0x0001f134 in module_setup (moduleq8445576, n94f4c) at eval.c:3293
#189 0x0001a4b0 in rb_eval (selfq8183648, n2ace5640) at eval.c:3180
#190 0x00021588 in rb_load (fnameq8403716, wrapa7468) at eval.c:5284
#191 0x000217fc in rb_f_load (argc argv2ace5640) at eval.c:5333
#192 0x00012a38 in call_cfunc (func217c8 <rb_f_load>, recvq8187688, len!47379076, 
    argc!47339264, argv7fffe93c) at eval.c:4285
#193 0x0001aac8 in rb_call0 (klassq8186288, recvq8187688, id9, argc argv7fffe93c, 
    body2acddc24, flags at eval.c:4423
#194 0x0001b3d0 in rb_call (klassq8186288, recvq8187688, mid9, argc argv7fffe93c, 
    scope at eval.c:4654
#195 0x00018a78 in rb_eval (selfq8445576, n2ace5640) at ruby.h:574
#196 0x0001af54 in rb_call0 (klassq8444076, recvq8445576, id7, argc argv7ffff020, 
    body2acd7fcc, flags at eval.c:4560
#197 0x0001b3d0 in rb_call (klassq8444076, recvq8445576, mid7, argc argv7ffff01c, 
    scope at eval.c:4654
#198 0x00018a78 in rb_eval (selfq8445576, n2ace5640) at ruby.h:574
#199 0x0001af54 in rb_call0 (klassq8444076, recvq8445576, id9, argc argv0, 
    body2acd70a4, flags at eval.c:4560
#200 0x0001b3d0 in rb_call (klassq8444076, recvq8445576, mid9, argc argv0, 
    scope at eval.c:4654
#201 0x00018a78 in rb_eval (selfq8183648, n2ace5640) at ruby.h:574
#202 0x00021f88 in ruby_run () at eval.c:1235
#203 0x00010900 in main (argc0 argv7ffffd24, envp7ffdcc00) at main.c:50

--------------030706030406040700000602--