--Apple-Mail-12--324519395
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset-ascii

You have introduced something called a "root node" without defining it. What do you mean by this?

I'm assuming here you mean that in your case, if you allocate the object in the script body, then set the local to nil, you can observe that the object appears to not be collected. As has been stated in the thread already, this is an artifact of the conservative GC. Even though you have set the local to nil, a reference to the object may still remain on the C stack. That reference can't be seen by ruby code because it is in stack memory that gcc setup and didn't clear when the value wasn't needed anymore.

This is unfortunate, but not the end of the world. It doesn't happen with every object allocated in a script body, only sometimes. The patch set you were pointed to goes to lengths to clear the stack space as much as it can so that there are none of these phantom references to confuse the GC. It does this by breaking up the main eval function into smaller functions (allowing stack space to be allocated and deallocated within the eval itself) and forcibly clearing the stack with memset.

I hope this clears it up.

 - Evan

On Aug 27, 2010, at 8:22 AM, Asher wrote:

> I very much appreciate the response, and this is helpful in describing the narrative, but it's still a few steps behind my question - but it may very well have clarified some points that help us get there. 
> 
> Let's stick with the example: a local variable is set as a reference to an object, the local variable is then set to nil so there is no longer a live reference to the object. No other ruby space commands have gone on, so unless Ruby is keeping junk behind the scenes, there should be no references - not on the Ruby stack, not on the C stack. How does this object get collected? As shown by the example, it is missed during the next attempt to GC (as well as any repeated attempts at this point).  
> So what will change to make that object collectable? Are you suggesting that because it is in Ruby's root node that it gets treated such that it won't be GC'd until the program terminates? 
> 
> Let's assume this is the case. I should therefore be able to write a script that creates a non-root object as a child to another object inside method scope, allow that method to go out of scope, and expect that the object will be GC'd (as it is neither a root node nor does it have any live references). 
> 
> This seems to be validated by the following Ruby code:
> 
>> require 'pp'
>> 
>> class Hash::Weak < Hash
>>   
>>   def []( key )
>> 
>>     # get the stored ID - a FixNum, not an object reference to our weak-referenced object
>>     obj_id = super( key.to_sym )
>> 
>>     # theoretically this should cause non-referenced objects to get cleaned up 
>>     # so long as nothing looks like a pointer or reference to it
>>     ObjectSpace.garbage_collect
>> 
>>     # now get our object from ID
>>     # if it had no references it should have been GC'd and we should get an 
>>     # rb_eRangeError "is not id value" (expected) or "is recycled object" (possible)
>>     obj = ObjectSpace._id2ref( obj_id )
>> 
>>     return obj
>> 
>>   end
>>   
>>   def []=( key, object )
>> 
>>     # FixNum have a constant ID for value, so can't be copied and can't be garbage collected
>>     # so object.__id__ cannot be a reference to a child of object and therefore cannot prevent
>>     # garbage collection on the object
>>     super( key.to_sym, object.__id__ )
>> 
>>   end
>>   
>> end
>> 
>> ##################################################
>> 
>> # non-rootnode demo
>> 
>> $weak_hash = Hash::Weak.new
>> 
>> class TestClass
>>   def test_method
>>     child_test_object = Object.new
>>     puts 'storing test object'
>>     $weak_hash[ :key ] = child_test_object
>>     puts 'hash now contains object id: ' + $weak_hash.pretty_inspect  20
>>   end
>> end
>> test_object	=	TestClass.new
>> 
>> test_object.test_method
>> puts 'id in hash should no longer be valid, as it is out of scope: '
>> invalid_key = $weak_hash[ :key ]
>> pp invalid_key
> 
> Output: 
> 
>> storing test object
>> hash now contains object id: {:key=>2160173880}
>> id in hash should no longer be valid, as it is out of scope:
>> /Users/ahaig/Projects/rp/ruby/weakhash/projects/RPWeakHash/weakhash.rb:21:in `_id2ref': 0x00000080c1a338 is recycled object (RangeError)	
>> from /Users/ahaig/Projects/rp/ruby/weakhash/projects/RPWeakHash/weakhash.rb:21:in `[]'
>> from /Users/ahaig/Projects/rp/ruby/weakhash/projects/RPWeakHash/weakhash.rb:56:in `<main>'
> 
> So that works as expected (so long as GC is run manually; if not run, object is obviously still valid). 
> 
> So let's try the same thing in this example that we did with the root node example:
> 
>> <weak hash code identical to above, so omitted>
>> 
>> # non-rootnode demo with nil-setting
>> 
>> $weak_hash = Hash::Weak.new
>> 
>> class TestClass
>>   def test_method
>>     child_test_object = Object.new
>>     puts 'storing test object'
>>     $weak_hash[ :key ] = child_test_object
>>     puts 'hash now contains object id: ' + $weak_hash.pretty_inspect  20
>> 
>>     puts 'setting variable referring to test object (ID: ' + child_test_object.__id__.to_s + ') to nil'
>>     child_test_object = nil
>>     puts 'ID for variable referring to test object is now: ' + child_test_object.__id__.to_s
>> 
>>     print 'getting test object (should fail with rb_eRangeError): '
>>     invalid_key = $weak_hash[ :key ]
>>     pp invalid_key
>>   end
>> end
>> test_object	=	TestClass.new
>> 
>> test_object.test_method
>> puts 'id in hash should no longer be valid, as it is out of scope: '
>> invalid_key = $weak_hash[ :key ]
>> pp invalid_key
> 
> Output: 
> 
>> storing test object
>> hash now contains object id: {:key=>2160328280}
>> setting variable referring to test object (ID: 2160328280) to nil
>> ID for variable referring to test object is now: 4
>> getting test object (should fail with rb_eRangeError):
>> /Users/ahaig/Projects/rp/ruby/weakhash/projects/RPWeakHash/weakhash.rb:21:in `_id2ref': 0x00000080c3fe58 is recycled object (RangeError)
>> 	
>> from /Users/ahaig/Projects/rp/ruby/weakhash/projects/RPWeakHash/weakhash.rb:21:in `[]'
>> 	
>> from /Users/ahaig/Projects/rp/ruby/weakhash/projects/RPWeakHash/weakhash.rb:56:in `test_method'
>> 	
>> from /Users/ahaig/Projects/rp/ruby/weakhash/projects/RPWeakHash/weakhash.rb:62:in `<main>'
> 
> So that also works as expected. 
> 
> It seems the only time that it does not work as expected, then, is when the object was instantiated with a reference in the root node. 
> 
> Of course, that is one of the most likely places for people to instantiate a reference. So can anything be done about this? Are we simply doomed to wait until program termination for any objects allocated in the root node to disappear? 
> 
> I intend to look into the patch suggested by brabuhr / gmail.com (https://sites.google.com/site/brentsrubypatches/), which (so far as this issue is concerned) appears to amount to:
> 
>> VALUE *rb_gc_stack_end = (VALUE *)STACK_GROW_DIRECTION;
>> #define rb_gc_wipe_stack() {   		\
>>   VALUE *sp = alloca(0);         		\
>>   VALUE *end = rb_gc_stack_end;  \
>>   rb_gc_stack_end = sp;          		\
>>   __stack_zero(end, sp);   			\
> 
> And some other basic support. I will follow up on that once I have some time to experiment (particularly sense the patch is intended for 1.8.7 not 1.9.2). Any particular thoughts on this approach? Presumably there is some reason it has not been patched to do so? 
> 
> In any case, any thoughts on any of this will be much appreciated.
> 
> Asher

<snip>


--Apple-Mail-12--324519395
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset-ascii

<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">You have introduced something called a "root node" without defining it. What do you mean by this?<div><br></div><div>I'm assuming here you mean that in your case, if you allocate the object in the script body, then set the local to nil, you can observe that the object appears to not be collected. As has been stated in the thread already, this is an artifact of the conservative GC. Even though you have set the local to nil, a reference to the object may still remain on the C stack. That reference can't be seen by ruby code because it is in stack memory that gcc setup and didn't clear when the value wasn't needed anymore.</div><div><br></div><div>This is unfortunate, but not the end of the world. It doesn't happen with every object allocated in a script body, only sometimes. The patch set you were pointed to goes to lengths to clear the stack space as much as it can so that there are none of these phantom references to confuse the GC. It does this by breaking up the main eval function into smaller functions (allowing stack space to be allocated and deallocated within the eval itself) and forcibly clearing the stack with memset.</div><div><br></div><div>I hope this clears it up.</div><div><br></div><div>&nbsp;- Evan<br><div><br></div><div><div><div>On Aug 27, 2010, at 8:22 AM, Asher wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>I very much appreciate the response, and this is helpful in describing the narrative, but it's still a few steps behind my question - but it may very well have clarified some points that help us get there.&nbsp;</div><div><br></div><div>Let's stick with the example: a local variable is set as a reference to an object, the local variable is then set to nil so there is no longer a live reference to the object. No other ruby space commands have gone on, so unless Ruby is keeping junk behind the scenes, there should be no references - not on the Ruby stack, not on the C stack. How does this object get collected? As shown by the example, it is missed during the next attempt to GC (as well as any repeated attempts at this point).&nbsp;</div><div><br></div><div>So what will change to make that object collectable? Are you suggesting that because it is in Ruby's root node that it gets treated such that it won't be GC'd until the program terminates?&nbsp;</div><div><br></div><div>Let's assume this is the case. I should therefore be able to write a script that creates a non-root object as a child to another object inside method scope, allow that method to go out of scope, and expect that the object will be GC'd (as it is neither a root node nor does it have any live references).&nbsp;</div><div><br></div><div>This seems to be validated by the following Ruby code:</div><div><br></div><blockquote type="cite"><blockquote class="webkit-indent-blockquote" style="margin: 0 0 0 40px; border: none; padding: 0px;"><div><div>require 'pp'</div></div><div><div><br></div></div><div><div>class Hash::Weak &lt; Hash</div></div><div><div>&nbsp;&nbsp;</div></div><div><div>&nbsp;&nbsp;def []( key )</div></div><div><div><br></div></div><div><div>&nbsp;&nbsp; &nbsp;# get the stored ID - a FixNum, not an object reference to our weak-referenced object</div></div><div><div>&nbsp;&nbsp; &nbsp;obj_id = super( key.to_sym )</div></div><div><div><br></div></div><div><div>&nbsp;&nbsp; &nbsp;# theoretically this should cause non-referenced objects to get cleaned up&nbsp;</div></div><div><div>&nbsp;&nbsp; &nbsp;# so long as nothing looks like a pointer or reference to it</div></div><div><div>&nbsp;&nbsp; &nbsp;ObjectSpace.garbage_collect</div></div><div><div><br></div></div><div><div>&nbsp;&nbsp; &nbsp;# now get our object from ID</div></div><div><div>&nbsp;&nbsp; &nbsp;# if it had no references it should have been GC'd and we should get an&nbsp;</div></div><div><div>&nbsp;&nbsp; &nbsp;# rb_eRangeError "is not id value" (expected) or "is recycled object" (possible)</div></div><div><div>&nbsp;&nbsp; &nbsp;obj = ObjectSpace._id2ref( obj_id )</div></div><div><div><br></div></div><div><div>&nbsp;&nbsp; &nbsp;return obj</div></div><div><div><br></div></div><div><div>&nbsp;&nbsp;end</div></div><div><div>&nbsp;&nbsp;</div></div><div><div>&nbsp;&nbsp;def []=( key, object )</div></div><div><div><br></div></div><div><div>&nbsp;&nbsp; &nbsp;# FixNum have a constant ID for value, so can't be copied and can't be garbage collected</div></div><div><div>&nbsp;&nbsp; &nbsp;# so object.__id__ cannot be a reference to a child of object and therefore cannot prevent</div></div><div><div>&nbsp;&nbsp; &nbsp;# garbage collection on the object</div></div><div><div>&nbsp;&nbsp; &nbsp;super( key.to_sym, object.__id__ )</div></div><div><div><br></div></div><div><div>&nbsp;&nbsp;end</div></div><div><div>&nbsp;&nbsp;</div></div><div><div>end</div></div><div><div><br></div></div><div><div>##################################################</div></div><div><br></div><div><div># non-rootnode demo</div></div><div><div><br></div></div><div><div>$weak_hash = Hash::Weak.new</div></div><div><div><br></div></div><div><div>class TestClass</div></div><div><div>&nbsp;&nbsp;def test_method</div></div><div><div>&nbsp;&nbsp; &nbsp;child_test_object = Object.new</div></div><div><div>&nbsp;&nbsp; &nbsp;puts 'storing test object'</div></div><div><div>&nbsp;&nbsp; &nbsp;$weak_hash[ :key ] = child_test_object</div></div><div><div>&nbsp;&nbsp; &nbsp;puts 'hash now contains object id: ' + $weak_hash.pretty_inspect &nbsp; &nbsp;</div></div><div><div>&nbsp;&nbsp;end</div></div><div><div>end</div></div><div><div>test_object<span class="Apple-tab-span" style="white-space:pre">	</span>=<span class="Apple-tab-span" style="white-space:pre">	</span>TestClass.new</div></div><div><div><br></div></div><div><div>test_object.test_method</div></div><div><div>puts 'id in hash should no longer be valid, as it is out of scope: '</div></div><div><div>invalid_key = $weak_hash[ :key ]</div></div><div><div>pp invalid_key</div></div></blockquote></blockquote><div><br></div><div>Output:&nbsp;</div><div><span class="Apple-style-span" style="font-family: 'Trebuchet MS', Helvetica, sans-serif; font-size: 13px; "></span></div><div><span class="Apple-style-span" style="font-family:Trebuchet MS', Helvetica, sans-serif; font-size: 13px; "></span></div><div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 12px/normal Helvetica; "><span class="Apple-style-span" style="font-size: medium;"><br></span></div></div><blockquote type="cite"><blockquote class="webkit-indent-blockquote" style="margin: 0 0 0 40px; border: none; padding: 0px;"><div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 12px/normal Helvetica; ">storing test object</div></div><div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 12px/normal Helvetica; ">hash now contains object id: {:key=&gt;2160173880}</div></div><div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 12px/normal Helvetica; ">id in hash should no longer be valid, as it is out of scope:</div></div><div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 12px/normal Helvetica; ">/Users/ahaig/Projects/rp/ruby/weakhash/projects/RPWeakHash/weakhash.rb:21:in `_id2ref': 0x00000080c1a338 is recycled object (RangeError)<span class="Apple-tab-span" style="white-space:pre">	</span></div></div><div>from /Users/ahaig/Projects/rp/ruby/weakhash/projects/RPWeakHash/weakhash.rb:21:in `[]'</div><div>from /Users/ahaig/Projects/rp/ruby/weakhash/projects/RPWeakHash/weakhash.rb:56:in `&lt;main&gt;'</div></blockquote></blockquote><div><br></div><div>So that works as expected (so long as GC is run manually; if not run, object is obviously still valid).&nbsp;</div><div><br></div><div>So let's try the same thing in this example that we did with the root node example:</div><div><br></div><blockquote type="cite"><blockquote class="webkit-indent-blockquote" style="margin: 0 0 0 40px; border: none; padding: 0px;"><div>&lt;weak hash code identical to above, so omitted&gt;</div><div><br></div><div><div># non-rootnode demo with nil-setting</div></div><div><div><br></div></div><div><div>$weak_hash = Hash::Weak.new</div></div><div><div><br></div></div><div><div>class TestClass</div></div><div><div>&nbsp;&nbsp;def test_method</div></div><div><div>&nbsp;&nbsp; &nbsp;child_test_object = Object.new</div></div><div><div>&nbsp;&nbsp; &nbsp;puts 'storing test object'</div></div><div><div>&nbsp;&nbsp; &nbsp;$weak_hash[ :key ] = child_test_object</div></div><div><div>&nbsp;&nbsp; &nbsp;puts 'hash now contains object id: ' + $weak_hash.pretty_inspect &nbsp; &nbsp;</div></div><div><div><br></div></div><div><div>&nbsp;&nbsp; &nbsp;puts 'setting variable referring to test object (ID: ' + child_test_object.__id__.to_s + ') to nil'</div></div><div><div>&nbsp;&nbsp; &nbsp;child_test_object = nil</div></div><div><div>&nbsp;&nbsp; &nbsp;puts 'ID for variable referring to test object is now: ' + child_test_object.__id__.to_s</div></div><div><div><br></div></div><div><div>&nbsp;&nbsp; &nbsp;print 'getting test object (should fail with rb_eRangeError): '</div></div><div><div>&nbsp;&nbsp; &nbsp;invalid_key =weak_hash[ :key ]</div></div><div><div>&nbsp;&nbsp; &nbsp;pp invalid_key</div></div><div><div>&nbsp;&nbsp;end</div></div><div><div>end</div></div><div><div>test_object<span class="Apple-tab-span" style="white-space:pre">	</span>=<span class="Apple-tab-span" style="white-space:pre">	</span>TestClass.new</div></div><div><div><br></div></div><div><div>test_object.test_method</div></div><div><div>puts 'id in hash should no longer be valid, as it is out of scope: '</div></div><div><div>invalid_key = $weak_hash[ :key ]</div></div><div><div>pp invalid_key</div></div></blockquote></blockquote><div><br></div>Output:&nbsp;<div><br><blockquote type="cite"><blockquote class="webkit-indent-blockquote" style="margin: 0 0 0 40px; border: none; padding: 0px;"><div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 12px/normal Helvetica; ">storing test object</div></div><div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 12px/normal Helvetica; ">hash now contains object id: {:key=&gt;2160328280}</div></div><div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 12px/normal Helvetica; ">setting variable referring to test object (ID: 2160328280) to nil</div></div><div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 12px/normal Helvetica; ">ID for variable referring to test object is now: 4</div></div><div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 12px/normal Helvetica; ">getting test object (should fail with rb_eRangeError):</div></div><div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 12px/normal Helvetica; ">/Users/ahaig/Projects/rp/ruby/weakhash/projects/RPWeakHash/weakhash.rb:21:in `_id2ref': 0x00000080c3fe58 is recycled object (RangeError)</div></div><div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 12px/normal Helvetica; "><span class="Apple-tab-span" style="white-space: pre; ">	</span></div>from /Users/ahaig/Projects/rp/ruby/weakhash/projects/RPWeakHash/weakhash.rb:21:in `[]'</div><div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 12px/normal Helvetica; "><span class="Apple-tab-span" style="white-space: pre; ">	</span></div>from /Users/ahaig/Projects/rp/ruby/weakhash/projects/RPWeakHash/weakhash.rb:56:in `test_method'</div><div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 12px/normal Helvetica; "><span class="Apple-tab-span" style="white-space: pre; ">	</span></div>from /Users/ahaig/Projects/rp/ruby/weakhash/projects/RPWeakHash/weakhash.rb:62:in `&lt;main&gt;'</div></blockquote></blockquote><div><br></div><div>So that also works as expected.&nbsp;</div><div><br></div><div>It seems the only time that it does not work as expected, then, is when the object was instantiated with a reference in the root node.&nbsp;</div><div><br></div><div>Of course, that is one of the most likely places for people to instantiate a reference. So can anything be done about this? Are we simply doomed to wait until program termination for any objects allocated in the root node to disappear?&nbsp;</div><div><br></div><div>I intend to look into the patch suggested by brabuhr / gmail.com (https://sites.google.com/site/brentsrubypatches/), which (so far as this issue is concerned) appears to amount to:</div></div><div><br></div><blockquote type="cite"><blockquote class="webkit-indent-blockquote" style="margin: 0 0 0 40px; border: none; padding: 0px;"><div><div><div>VALUE *rb_gc_stack_end = (VALUE *)STACK_GROW_DIRECTION;</div></div></div><div><div><div>#define rb_gc_wipe_stack() { &nbsp; <span class="Apple-tab-span" style="white-space:pre">		</span>\</div></div></div><div><div><div>&nbsp;&nbsp;VALUE *sp = alloca(0); &nbsp; &nbsp; &nbsp; &nbsp; <span class="Apple-tab-span" style="white-space:pre">		</span>\</div></div></div><div><div><div>&nbsp;&nbsp;VALUE *end = rb_gc_stack_end; &nbsp;\</div></div></div><div><div><div>&nbsp;&nbsp;rb_gc_stack_end = sp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;<span class="Apple-tab-span" style="white-space:pre">		</span>\</div></div></div><div><div><div>&nbsp;&nbsp;__stack_zero(end, sp); &nbsp; <span class="Apple-tab-span" style="white-space:pre">		/span>\<span></span></div></div></div></blockquote></blockquote><div><div><br></div><div>And some other basic support. I will follow up on that once I have some time to experiment (particularly sense the patch is intended for 1.8.7 not 1.9.2). Any particular thoughts on this approach? Presumably there is some reason it has not been patched to do so?&nbsp;</div><div><br></div><div>In any case, any thoughts on any of this will be much appreciated.</div><div><br></div><div>Asher</div></div></div></blockquote><br></div><div>&lt;snip&gt;</div><br></div></div></body></html>-Apple-Mail-12--324519395--