Issue #13901 has been updated by marcandre (Marc-Andre Lafortune).


My friend Maxime Lapointe and I have been hacking on a pure Ruby gem called °»DeepCover°… to do branch/method/everything coverage, so we thought we should post some info here for feedback.

We are working actively on it, but the base is written and functional, so it°«s probably time to start introducing it to the community at large: https://github.com/deep-cover/deep-cover

As an example, here's the coverage we can produce for
ActiveSupport: https://deep-cover.github.io/rails-cover/activesupport/
ActiveRecord: https://deep-cover.github.io/rails-cover/activerecord/

Note: we currently use `istanbul` for output (far from perfect), but are working on a much nicer direct HTML output.

An example of expression that the builtin coverage doesn°«t (yet) detect is line 190 of https://deep-cover.github.io/rails-cover/activerecord/lib/active_record/schema_dumper.rb.html

DeepCover is based on the awesome `parser` gem. We instrument any Ruby code such that we can know, for any node, how many time it°«s been executed. More precisely, for any node, we know how many times control flow has "entered" the node, how many times it has "exited" the node normally, and by deduction how many times control flow has been interrupted (raise, throw, return, next, etc...).

Since it's pure Ruby, it is very easy to then customize the analysis results. For example it is possible to require coverage of implicit `else` in a `case` statement or not, for default arguments to be covered too (or not, or only if they're not simple literals), or allow an uncovered `raise` to be ignored. It's also compatible with Ruby 2.0+ and JRuby.

We aim to provide a powerful API, as well as one that can replace MRI°«s while maintaining compatibility.

If experienced Rubyists want to delve into it and contribute, this would be awesome. Please contact us so we can coordinate the efforts.


----------------------------------------
Feature #13901: Add branch coverage 
https://bugs.ruby-lang.org/issues/13901#change-67676

* Author: mame (Yusuke Endoh)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: 
----------------------------------------
I plan to add "branch coverage" (and "method coverage") as new target types of coverage.so, the coverage measurement library.  I'd like to introduce this feature for Ruby 2.5.0.  Let me to hear your opinions.

## Basic Usage of the Coverage API

The sequence is the same as the current: (1) require "coverage.so", (2) start coverage measurement by `Coverage.start`, (3) load a program being measured (typically, a test runner program), and (4) get the result by `Coverage.result`.

When you pass to `Coverage.start` with keyword argument "`branches: true`", branch coverage measurement is enabled.

test.rb

~~~
require "coverage"
Coverage.start(lines: true, branches: true)
load "target.rb"
p Coverage.result
~~~

target.rb

~~~
1: if 1 == 0
2:   p :match
3: else
4:   p :not_match
5: end
~~~

By measuring coverage of target.rb, the result will be output (manually formatted):

~~~
$ ruby test.rb
:not_match
{".../target.rb" => {
  :lines => [1, 0, nil, 1, nil],
  :branches => {
    [:if, 0, 1] => { [:then, 1, 2] => 0, [:else, 2, 4] => 1 }
  }
}
~~~

`[:if, 0, 1]` reads "if branch at Line 1", and `[:then, 1, 2]` reads "then clause at Line 2".  So, `[:if,0,1] => { [:then,1,2]=>0, [:else,2,4]=>0 }` reads "the branch from Line 1 to Line 2 has never executed, and the branch from Line 1 to Line 4 has executed once."

The second number (`0` of `[:if, 0, 1]`) is a unique ID to avoid conflict, just in case where multiple branches are written in one line.  This format of a key is discussed in "Key format" section.

## Why needed

Traditional coverage (line coverage) misses a branch in one line.  Branch coverage is useful to find such untested code.  See the following example.

target.rb

~~~
p(:foo) unless 1 == 0
p(1 == 0 ? :foo : :bar)
~~~

The result is:

~~~
{".../target.rb" => {
  :lines => [1, 1],
  :branches => {
    [:unless, 0, 1] => { [:else, 1, 1] => 0, [:then, 2, 1] => 1 },
    [:if, 3, 2] => { [:then, 4, 2] => 0, [:else, 5, 2] => 1 }
  }
}}
~~~

Line coverage tells coverage 100%, but branch coverage shows that the `unless` statement of the first line has never taken true and that the ternary operator has never taken true.

## Current status

I've already committed the feature in trunk as an experimental feature.  To enable the feature, you need to set the environment variable `COVERAGE_EXPERIMENTAL_MODE` = `true`.  I plan to activate this feature by default by Ruby 2.5 release, if there is no big problem.

## Key format

The current implementation uses `[<label>, <unique ID>, <lineno>]`, like `[:if, 0, 1]`, to represent the site of branch.  `<unique ID>` is required for the case where multiple branches are in one line.

I think this format is arguable.  I thought of some other candidates:

* `[<label>, <lineno>, <column-no>]`: A big problem, how should we handle TAB character?
* `[<label>, <offset from file head>]`: Looks good for machine readability.
* Are `<label>` and `<lineno>` needed? They are useful for human, but normally, this result will be processed by a visualization script (such as SimpleCov).

What do you think?  I'm now thinking that `[<label>, <lineno>, <offset from file head>]` is reasonable, but it is hard for me to implement.  I'll try later but parse.y is so difficult... (A patch is welcome!)

## Compatibility

This API is 100% compatible.  If no keyword argument is given, the result will be old format, i.e., a hash from filename to an array that represents line coverage.

~~~
# compatiblie mode
Coverage.start
load "target.rb"
p Coverage.result
#=> {".../target.rb" => [1, 1, 1, ...] }

# If "lines: true" is given, the result format differs slightly
Coverage.start(lines: true)
load "target.rb"
p Coverage.result
#=> {".../target.rb" => { :lines => [1, 1, 1, ...] } }
~~~

## Method coverage

Method coverage is also supported.  You can measure it by using `Coverage.start(methods: true)`.

target.rb

~~~
 1: def foo
 2: end
 3: def bar
 4: end
 5: def baz
 6: end
 7:
 8: foo
 9: foo
10: bar
~~~

result (manually formatted)

~~~
{".../target.rb"=> {
  :methods => {
    [:foo, 0, 1] => 2,
    [:bar, 1, 3] => 1,
    [:baz, 2, 5] => 0,
  }
}}
~~~

## Notes

* `if` statements whose condtion is literal, such as `if true` and `if false`, are not considered as a branch.

* `while`, `until`, and `case` are also supported.  See Examples 2 and 3.

* This proposal is based on #9508.  The proposal has [some spec-level issues](https://github.com/ruby/ruby/pull/511#issuecomment-328753499), but the work was really inspiring me.

## Future work

* Someone may want to know how many times an one-line block is executed, such as `n.times {  }`.

* Someone may want to know how many times each method call is executed, such as `obj.foo.bar.baz` (For example, if method `foo` always raises an exception, calls to `bar` and `baz` are not executed.)

## Some examples

### Example 1

target.rb

~~~
1: if 1 == 0
2:   p :match1
3: elsif 1 == 0
4:   p :match2
5: else
6:   p :not_match
7: end
~~~

result (manually formatted)

~~~
{"target.rb" => {
  :lines => [1, 0, 1, 0, nil, 1, nil],
  :branches => {
    [:if, 1] => { [:then, 2] => 0, [:else, 3] => 1 },
    [:if, 3] => { [:then, 4] => 0, [:else, 5] => 1 },
  }
}
~~~

### Example 2

target.rb

~~~
 1:case :BOO
 2:when :foo then p :foo
 3:when :bar then p :bar
 4:when :baz then p :baz
 5:else p :other
 6:end
 7:
 8:x = 3
 9:case
10:when x == 0 then p :foo
11:when x == 1 then p :bar
12:when x == 2 then p :baz
13:else p :other
14:end
~~~

result (manually formatted)

~~~
{"target.rb" => {
  :lines => [1, 0, 0, 0, 1, nil, nil, 1, 1, 1, 1, 1, 1, nil],
  :branches => {
    [:case, 1] => {
      [:when, 2] => 0,
      [:when, 3] => 0,
      [:when, 4] => 0,
      [:else, 5] => 1
    },
    [:case, 8] => {
      [:when, 9] => 0,
      [:when, 10] => 0,
      [:when, 11] => 0,
      [:else, 12] => 1
    }
  }
}
~~~

### Example 3

target.rb

~~~
 1:n = 0
 2:while n < 100
 3:  n += 1
 4:end
~~~

result (manually formatted)

~~~
{"target.rb" => {
  :lines => [1, 101, 100, nil],
  :branches => {
    [:while, 2] => {
      [:body, 3] => 100,
      [:end, 5] => 1
    }
  }
}
~~~



-- 
https://bugs.ruby-lang.org/

Unsubscribe: <mailto:ruby-core-request / ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-core>