meta: task dump stabilization

[jswrenn]

Feature Request: #5457

PRs:

• Implement task dumps for current-thread runtime. #5608
• taskdump: implement task dumps for multi-thread runtime #5717

Roadmap:

• Runtime Support
  • Implement task dumps for current-thread runtime. (#5608)
    • Support capturing task dump of current-thread runtime from outside runtime context.
  • Implement task dumps for multi-thread runtime. (#5717)
  • Ensure no fields added to runitme::Context do not have Drop implementations.
• Trace Quality Improvements
  • Instrument all tokio leaves. (#5708)
  • Instrument frames with extra metadata (e.g., function arguments).
  • Provide a public API for instrumenting frames.
  • Increase platform support.
  • Maintain insertion order of frames in trace output.
• Task Dump Improvements
  • Include additional task metadata:
    • spawn location
    • name
    • ID (runtime: include task Id in taskdumps #6328)
  • Ensure that FuturesUnordered and LocalSet futures are included in dumps.
  • Add functionality for dumping subset of tasks.
  • Include Very Stuck Tasks
• Remove External Dependencies
  • Remove backtrace-rs dependency.
• Misc
  • Clean up #[cfg]s.

[carllerche]

@jswrenn Something I noticed (and I updated the checklist).

The trace::Context field added to runtime::Context has a Vec, which has a drop implementation. We need to make sure that runtime::Context does not require a drop implementation to take advantage of const thread locals.

I know scheduler::Handle currently requires one, but I have a plan to remove that.

[scottlamb]

Include Very Stuck Tasks

What's the current behavior with Very Stuck Tasks? I'm trying to use task dumps to diagnose some tasks not making progress. No luck so far. Besides sometimes getting panics (#6185), I'm seeing tasks where Task::trace returns the empty string. I'm wondering if these are Very Stuck Tasks (and trying to figure out my next step to tracking down what's happening). Really wishing I had the trace for these, or at least the "additional task metadata".

[jswrenn]

Thanks for reporting the panic issue — I had really hoped we fixed that, but I guess not. I'll look into that ASAP.

I believe the behavior of taskdumps in the presence of very-stuck-tasks should be to hang. Tasks dumps work by waiting for all polls to complete, then re-polling all tasks in a special tracing mode. If a poll never completes, this synchronization point cannot be reached and a taskdump cannot be produced. (That's not ideal, but an infinite-looping poll is where a traditional debugger becomes the better tool of choice, anyways.)

I would only expect task dumping to produce an empty string in cases when the dumped runtime has no work to do. This might happen if you have multiple runtimes and are polling one that happens to have no work to do.

That said, the fact that you're running into #6185 means that some of my assumptions are wrong.

[scottlamb]

I would only expect task dumping to produce an empty string in cases when the dumped runtime has no work to do. This might happen if you have multiple runtimes and are polling one that happens to have no work to do.

Hmm. I do have multiple runtimes (I give quinn connections their own single-threaded runtime due to not-yet-understood quinn-rs/quinn#1433), but I'm only dumping the main multi-threaded runtime, and it's returning a bunch of tasks, some with meaningful traces and some with empty strings. I'm assuming the other runtimes' tasks wouldn't show up in this runtime's dump at all?

[jswrenn]

I think empty strings could also arise if the tasks are simple and aggressively inlined. Are you experiencing this in debug builds?

I'm assuming the other runtimes' tasks wouldn't show up in this runtime's dump at all?

Correct.