Issue #16027 has been updated by duerst (Martin D=FCrst).


@dalehamel: Do you have any parts implemented already, or any plans for imp=
lementation? Would you be ready to do significant implementation work if th=
is feature got accepted? What parts do you think could be done as a Gem, an=
d what parts would need to be part of the Ruby implementation core? How cou=
ld this be made more OS-independent (besides Linux, Darvin, and Windows, th=
ere are also various BSD flavors,...).

----------------------------------------
Feature #16027: Update Ruby's dtrace / USDT API to match what is exposed vi=
a the TracePoint API
https://bugs.ruby-lang.org/issues/16027#change-80152

* Author: dalehamel (Dale Hamel)
* Status: Open
* Priority: Normal
* Assignee: =

* Target version: =

----------------------------------------
# Abstract

I propose that Ruby's "dtrace" support be extended to match what is availab=
le in the TracePoint API, as was the case until feature [Feature #15289] la=
nded.

# Background

I will refer to Ruby's "dtrace" bindings as USDT bindings for simplicity, a=
s this is the typo of dtrace probe that they support.

Prior to [Feature #15289] being merged, Ruby's tracepoint API was able to t=
race only 'all' instances of a type of event.

Ruby added support for tracing ruby with dtrace, and so Ruby's USDT Ruby Tr=
acePoint API were "in sync".

Once the Ruby TracePoint API recently added the ability to do filtered trac=
ing in [Feature #15289], it added new functionality but brought the TracePo=
int and USDT API out of sync.

Currently the TracePoint API is ahead of the USDT API, which presents the p=
roblem. There is valuable debug information available, but we do not have
a way to access it with dtrace instrumentation.

Additionally, the recent release of bpftrace adds support for USDT tracing =
on linux, which makes this a valuable opportunity to be able to use Ruby's =
TracePoint API in an efficient and targeted way for production tracing. To =
achieve this, we must synchronize the features of the USDT and TracePoint A=
PI.

What is currently lacking is the ability to do filtered, selective tracing =
as the `TracePoint#enable` call now supports as per [prelude.rb#L141](https=
://github.com/ruby/ruby/blob/master/prelude.rb#L141)

# Proposal

When enabling a TracePoint, users can specify a flag: `usdt: [LIST_OF_SIMPL=
E_TYPES]`, which will trigger Ruby to also enable the USDT API for when it =
enables TracePoints.

Within the TracePoint block, users can call `tp.fire` to send USDT data. So=
 the new default API is:

```ruby
trace.enable(target: nil, target_line: nil, target_thread: nil: usdt: nil)
```

And the usage might look like:

```ruby
trace.enable(target: method(:foo), target_line: 5, usdt: [Integer, String])=
 do |tp|
  tp.fire(tp.lineno, "Any String I want to send")
end
```

The types specified must be simple types such as `Integer` or `String`, giv=
en by their names as constants. When data to the tracepoint, the types must=
 match. If they don't, the tracer won't be able to interpret them properly,=
 but nothing should crash.

# Details

I propose that Ruby optionally generate ELF (Linux) or DOF (Darwin) annotat=
ions for TracePoint targets when they are enabled.

As ruby is a dynamic language, it cannot do this natively (yet) though Ruby=
 JIT may make this easier, but for now it is not suitable for production us=
e.

To get around this, Ruby can either generate the DOF or ELF stub shared lib=
rary itself, for example it may do one per class, treating the class as the=
 "provider" for the USDT API, and the methods as tracepoints. This is the a=
pproach used by [libusdt](https://github.com/chrisa/libusdt), which generat=
es DOF usable on Darwin, BSD, and other platforms, and [libstapsdt](https:/=
/github.com/sthima/libstapsdt), which generates ELF stubs for use on linux.

When a tracepoint is triggered, the user may be able to call a new API `Tra=
cePoint#fire`, to send data to the Kernel via the USDT API, using the gener=
ated ELF stub as a bridge, giving the kernel an address to target in order =
to receive this data.

Upon enabling a tracepoint, we can either generate these stubs internally, =
or by linking to an external library that must be enabled at configure time=
 (without this, USDT tracing wouldn't be enabled at all).

It may be possible to use the existing bridge that is used by ruby jit, or =
have an experimental flag such as `--usdt` that enables support for generat=
ing these stubs.

It may be more consistent with the future Ruby JIT to do this, or else Ruby=
 can generate these stubs by its own native code, but this will require a s=
ort of merging of libusdt and libstapsdt. This would add a dependency to th=
e libelf development header, but that is probably not a problem on Linux pl=
atforms.

I would suggest the first approach, if this feature is accepted, would be t=
o try and implement the ELF / DOF generation directly in Ruby. What libstap=
sdt and libusdt do isn't that complex and could be done in its own C file t=
hat probably wouldn't be too large.

Failing that approach, it may be worth investigating the Ruby JIT code to s=
ee if a compiler can generate these stubs for us easily. This approach woul=
d be to have ruby generate C code that results in the necessary DOF/ELF ann=
otations, and have the compiler pipeline used by ruby JIT to generate the f=
ile. This couples the feature to ruby jit though.

# Usecase

This feature would be used by dtrace / bpftrace users to debug ruby applica=
tions. It may be possible for other platforms to benefit from this too, but=
 I think the main use case is for Linux system administrators and developer=
s to use external debuggers (dtrace/bpftrace) to introspect Ruby's behavior.

# Discussion

## Pros:

* Syncs the Ruby TracePoint and USDT API
* Allows for much more dynamic and targeted USDT tracing
* Can help to find problems in both development and production
* Can be used for performance and error analysis
* Is better than printing, as emitting/collecting data is only done while a=
 "debugger is attached"

## Cons:

* Complexity introduced, in order to generate the ELF/DOF stub files
* Not easily ported to other platforms
* Isn't fully consistent with the current dtrace functionality of Ruby, whi=
ch is built-in to the VM

# Limitation

This will only work on *Nix platforms, and probably just on Linux to start,=
 as that is where most of the benefits are.

If the Ruby JIT approach is preferred or much simpler, then that functional=
ity will be tied to the Ruby JIT functionality.

# See also

* https://bpf.sh/usdt-report-doc/index.html a document describing my experi=
mental gem ruby-static-tracing, which prototypes this functionality outside=
 of the RubyVM
* https://bpf.sh/production-breakpoints-doc/index.html a work-in-progress o=
n adding more dynamic method and line based USDT tracing to ruby, built ato=
p ruby-static-tracing now using the ruby tracepoint API.




-- =

https://bugs.ruby-lang.org/

Unsubscribe: <mailto:ruby-core-request / ruby-lang.org?subject=3Dunsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-core>