kvstreamer: impact on go scheduling latency

[tbg]

Describe the problem

I was looking at some customer statement traces and noticed that frequently, kvstreamer would created several hundred goroutines at once. This leads to temporary scheduler overload

kvstreamer is in no way unique in causing such behavior, but it is one instance I've noticed multiple times. At present, I'm unable to quantify the performance impact in a concrete example, but I expect it to have substantial impact on the tail, especially in the case in which each individual kvstreamer request is computationally nontrivial.

To Reproduce

I did not reproduce this but saw it in a customer system, but I assume it is straightforward to reproduce and detect via an execution trace.

Note in the screenshot below how the scheduler suddenly finds itself with (at peak) 400+ runnable goroutines. This is a 32vcpu machine, so a single kvstreamer may launch 256 goroutines. These goroutines will compete for CPU time with other foreground reads that may be deemed more important by the user.

Unsure how to classify this (which A- label? Is SQL-Queries the right team?).

Jira issue: CRDB-41788

[mgartner]

@yuzefovich is it possible this is related to #134132?

[yuzefovich]

No, I think it's orthogonal to #134132 and something we haven't thought of before.