Virtual thread metrics

[shakuzen]

Follow-up from #3650. See https://openjdk.org/jeps/444#JDK-Flight-Recorder-JFR

Existing thread metrics do not cover virtual threads - ThreadMXBean only provides statistics on platform threads.
New potential instrumentation of virtual threads is only available via JFR. We can count started/finished virtual threads. The other events available are:

• jdk.VirtualThreadPinned indicates that a virtual thread was pinned (and its carrier thread wasn’t freed). This event is enabled by default with a threshold of 20 ms.
• jdk.VirtualThreadSubmitFailed indicates that starting or unparking a virtual thread failed, probably due to a resource issue. Parking a virtual thread releases the underlying carrier thread to do other work, and unparking a virtual thread schedules it to continue. This event is enabled by default.

[juliojgd]

Any developments here?

[jonatan-ivanov]

Not much I guess. Since this is not available through JMX (like metrics for platform threads are), we need to utilize JFR and jdk.VirtualThreadStart/jdk.VirtualThreadEnd events (disabled by default) which might be a bit more effort. Are you interested in submitting a PR?

[juliojgd]

I am not very familiar with JFR but, in order to retrieve the events count, shouldn't we start and stop a JFR recording? Or even keeping an active recording for the whole life of the JVM?

My knowledge about JFR is not enough to contribute this PR. Anyway I will try to find if in Java 22 or beyond there is any plan or JEP to enhance ThreadMXBean (or create a specific one for VT's) that could ease the retrieval of this measures.

[jonatan-ivanov]

I am not very familiar with JFR but, in order to retrieve the events count, shouldn't we start and stop a JFR recording?

I need to look into this but not necessarily in the classic sense where JFR is writing the data into a file. What we need is receiving the JFR events, we need to investigate how to do it so that it does not disturb the app.

Anyway I will try to find if in Java 22 or beyond there is any plan or JEP to enhance ThreadMXBean (or create a specific one for VT's) that could ease the retrieval of this measures.

Having this info coming from ThreadMXBean would be the best since we are already interacting with it for platform threads.

[ArtyomGabeev]

Hello here,

Let me know if you want me try and contribute support of VT metrics based on JFR continuous event listener.

Or are we waiting this data to be exposed via jmx?

[shakuzen]

We would consider an implementation based on the JFR events. I am not aware of any plans for anything to be added to JMX, so I don't think just waiting is necessarily likely to yield results. We could ask on the corresponding JDK mailing list about adding support for virtual thread metrics to JMX if there's a reason that's better over the JFR event route.

It would also be good for users to share what virtual thread metrics in particular they are interested in and why so we can understand what will be most helpful and why.

[ArtyomGabeev]

I think we may be interested in two JFR events:

1. VirtualThreadPinnedEvent - expose as a time, with pinning duration
2. VirtualThreadSubmitFailedEvent - expose as a counter

For the first one, things to consider:

1. Add a tag with stackframe where pinning happened? e.g.: com.test.ThreadPinningExample.sleep(Duration)
2. Additionally, I would like to be able to perform log once with extended strack trace, but I'm not sure this is a responsibility of micrometer library.

Here is an example of VirtualThreadPinnedEvent:

jdk.VirtualThreadPinned {
  startTime = 16:58:30.051 (2024-04-14)
  duration = 251 ms
  eventThread = "web-vt-729a03c1-5456-4686-90b2-b2659291752f" (javaThreadId = 69, virtual)
  stackTrace = [
    java.lang.VirtualThread.parkOnCarrierThread(boolean, long) line: 677
    java.lang.VirtualThread.parkNanos(long) line: 636
    java.lang.VirtualThread.sleepNanos(long) line: 793
    java.lang.Thread.sleep(long) line: 507
    com.test.ThreadPinningExample.sleep(Duration) line: 36
    ...
  ]
}

[ArtyomGabeev]

I've opened draft PR to demonstrate intentions and come up with the question:

I think this instrumentation should be specific to java-21. Should I go and create a java-21 source set or separate gradle module?