kvstreamer: super-linear increase in latency and number of gPRC calls in InOrder mode

[mgartner]

See the reproduction here: https://gist.github.com/mgartner/6163ac9079c64ca0d46592ff85f115ce

This reproduction runs 3 queries, index-joining ~10k, ~20k, and ~30k rows each. The latency and number of gRPC calls for each index-join grows super-linearly.

This seems related to #113729. That issue was addressed by #113809, however, I could not improve the performance of this new reproduction with any setting of streamer_in_order_eager_memory_usage_fraction - I tried as low as 0.05 and as high as 0.95.

I believe the code/comment here may be relevant:

cockroach/pkg/kv/kvclient/kvstreamer/streamer.go

Lines 1106 to 1127 in c83c57d

	// The next batch is not head-of-the-line, and the budget has used
	// up more than eagerMemUsageLimitBytes bytes. At this point, we
	// stop issuing "eager" requests, so we just exit.
	//
	// This exit will not lead to the streamer deadlocking because we
	// already have at least one batch to serve, and at some point we'll
	// get into this loop with headOfLine=true, so we'll always be
	// issuing at least one batch, eventually.
	//
	// This behavior is helpful to prevent pathological behavior
	// observed in #113729. Namely, if we issue too many batches eagerly
	// in the InOrder mode, the buffered responses might consume most of
	// our memory budget, and at some point we might regress to
	// processing requests one batch with a single Get / Scan request at
	// a time. We don't want to just drop already received responses
	// (we've already discarded the original singleRangeBatches anyway),
	// so this mechanism allows us to preserve a fraction of the budget
	// to processing head-of-the-line batches.
	//
	// Similar pattern can occur in the OutOfOrder mode too although the
	// degradation is not as severe as in the InOrder mode.
	return nil

Jira issue: CRDB-43434

[blathers-crl]

Hi @mgartner, please add branch-* labels to identify which branch(es) this C-bug affects.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[mgartner]

One workaround that may help improve performance in some cases is to disable the streamer with SET streamer_enabled = false;.