Issue #9572 has been updated by danmayer (Dan Mayer).

ruby -v changed from uby 2.0.0p353 (2013-11-22 revision 43784) [x86_64-darwin12.4.0] to ruby 2.6.0dev (2018-01-13 trunk 61811) [x86_64-darwin15]

I would like to work on closing this out but I believe due to changes in the API expectations since the bug was filed this is likely to not be considered a bug anymore but more likely to be considered to be expanding the feature set. I am thinking the best way to do this is to expand the public API of Coverage a bit.

I have detailed the bug and feature in this small repo to explore the interface: https://github.com/danmayer/coverage-bug

Details copied here to make sure they are in the Ruby system of record.

---

This shows a bug with Ruby's Coverage, as tracked on [ruby bug tracker #9572](https://bugs.ruby-lang.org/issues/9572).

to execute just run: `ruby ./example.rb`

This demonstrates why Coverage from the ruby std-lib isn't useful for the kind of data I want to get from coverband.

previous output (Ruby <= 2.1.1)

~~~ ruby
method_a called
{"/Users/danmayer/projects/coverage-bug/covercheck.rb"=>[1, 1, nil, nil, 1, 0, nil, nil, 1, 1, nil, 0, nil, 0, 0, 0, nil, nil, nil, nil, nil, nil, nil, nil, nil]}
this doesn't matter and is between coverage
method_b_called
{"/Users/danmayer/projects/coverage-bug/covercheck.rb"=>[]}
~~~

current output:

~~~ ruby
method_a called
{"/Users/danmayer/projects/coverage-bug/covercheck.rb"=>[1, 1, nil, nil, 1, 0, nil, nil, 1, 1, nil, 0, nil, 0, 0, 0, nil, nil, nil, nil, nil, nil, nil, nil, nil]}
this doesn't matter and is between coverage
method_b_called
{}
~~~

desired output:

~~~ ruby
method_a called
{"/Users/danmayer/projects/coverage-bug/covercheck.rb"=>[1, 1, nil, nil, 1, 0, nil, nil, 1, 1, nil, 0, nil, 0, 0, 0, nil, nil, nil, nil, nil, nil, nil, nil, nil]}
this doesn't matter and is between coverage
method_b_called
{"/Users/danmayer/projects/coverage-bug/covercheck.rb"=>[0, 0, nil, nil, 1, 1, nil, nil, 0, 0, nil, 0, nil, 1, 1, 1, nil, nil, nil, nil, nil, nil, nil, nil, nil]}
~~~

## Proposal

I would still like to have access to a re-entrant coverage, but given upstream chances since this initial bug, Ruby no longer returns `{'file.rb' => []}` as it was inaccurate to say the file had nothing. Currently, Ruby has removed all the files that were required prior to the initial coverage.start call. The method is re-entrant, but only works for **newly required** files.

I am thinking perhaps that functionality is intended and used for some use cases. It doesn't fit my use case or expectations. I am now considering a different approach to achieve what I would like. I think my needs would be better met with a new API with additional functionality for the Coverage object. This is similar to how `peek_result` was added to coverage. That didn't change the current expectations of any of the existing public API.

Thoughts are we could add the below methods to the public API:

~~~ ruby
Coverage.pause # removes event hooks, leaving data intact
Coverage.reset # leaves all data tracking, but resets counts to 0
Coverage.resume # adds back event hooks
~~~

Which could be used to collect coverage results on demand over time, enabling and disabling as needed. The example below illustrates basic usage

`example.rb`

~~~ ruby
require 'coverage'
Coverage.start
require './covercheck'
~~~

`covercheck.rb`

~~~ ruby
def method_a
  puts "method_a called"
end
 
def method_b
  puts "method_b_called"
end

method_a

puts Coverage.pause
puts Coverage.peek_result
puts Coverage.reset
 
puts "this line is run between coverage being enabled and shouldn't get tracked"
 
Coverage.resume
method_b
puts Coverage.result
~~~

The expected results would be something like below:

~~~ ruby
method_a called
{"/Users/danmayer/projects/coverage-bug/covercheck.rb"=>[1, 1, nil, nil, 1, 0, nil, nil, 1, 1, nil, 0, nil, 0, 0, 0]}
puts "this line is run between coverage being enabled and shouldn't get tracked"
method_b_called
{"/Users/danmayer/projects/coverage-bug/covercheck.rb"=>[0, 0, nil, nil, 1, 1, nil, nil, 1, 1, nil, 0, nil, 1, 1, 1]}
~~~

This could allow some extremely interesting code usage tracking, such as pausing and resuming in before / after hooks in Rails to track usage only in particular routes, controllers, modules, or etc. Using reset you could get just changes during a given run, or user could not reset the data and build up a results set over time, but still only incur the performance overhead on a very specific section of code.

I intend to be able to expand the capabilities of [Coverband](https://github.com/danmayer/coverband) using this feature as well as reduce the performance costs it currently incurs to run it.

----------------------------------------
Bug #9572: Restarting Coverage does not produce correct coverage result
https://bugs.ruby-lang.org/issues/9572#change-69571

* Author: sean_ferguson (Sean Ferguson)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: 
* ruby -v: ruby 2.6.0dev (2018-01-13 trunk 61811) [x86_64-darwin15]
* Backport: 1.9.3: UNKNOWN, 2.0.0: UNKNOWN, 2.1: UNKNOWN
----------------------------------------
Feature 4796(https://www.ruby-forum.com/topic/1811306#1001864) appears
to have a bug in it.

While the test listed there does pass the following test does not:
<pre>
require "test/unit"
require "coverage"
require 'tmpdir'
class TestCoverage < Test::Unit::TestCase
  def test_restarting_coverage
    Dir.mktmpdir {|tmp|
      Dir.chdir(tmp) {
        File.open("test.rb", "w") do |f|
          f.puts <<-EOS
            def coverage_test_method
              puts :ok
            end
          EOS
        end
        Coverage.start
        require tmp + '/test.rb'
        Coverage.result
        Coverage.start
        coverage_test_method
        result = Coverage.result
        assert_equal 1, result.size
        assert_equal [0, 1, nil], result.first[1] # coverage stats show an empty array here
      }
    }
  end
end
</pre>
It appears that while the coverage is finding the correct files it is
not giving any coverage stats for those files. Knowing this information would be very helpful in determining test coverage data for individual test files. I'm not very familiar
with how the coverage library works, but if you can point me at where to
look I can give fixing it a try.

Thanks,

Sean Ferguson



-- 
https://bugs.ruby-lang.org/

Unsubscribe: <mailto:ruby-core-request / ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-core>