rpc: count error responses towards batch response size limit

[handfuzzer]

Fixes #33814

Description

This PR fixes batch response size limit accounting for error responses.

Currently, (*handler).handleBatch only adds len(resp.Result) to responseBytes. For error responses, resp.Result is nil, so large error.data payloads are not counted and batch responses may exceed maxResponseSize.

Changes:

• rpc/handler.go: include error responses in responseBytes accounting.
• rpc: add regression test TestBatchResponseSizeLimitCountsErrorResponses.

Expected behaviour

When Server.SetBatchLimits(itemLimit, maxResponseSize) (or WithBatchResponseSizeLimit) is configured, the server should stop processing additional items in a batch once the cumulative response size exceeds maxResponseSize.

This limit should apply uniformly to both:

• Successful responses (result)
• Error responses (error, including error.data via rpc.DataError)

Once the limit is exceeded, subsequent batch items should return -32003 (response too large).

Actual behaviour

In rpc/handler.go, (*handler).handleBatch only accounts for len(resp.Result) when accumulating responseBytes.

For error responses:

• resp.Result is nil
• Large error payloads (especially large error.data) are not counted toward BatchResponseMaxSize

As a result, batch responses can significantly exceed maxResponseSize, and -32003 (response too large) is not reliably returned for later batch items.

---

Proposed fix

In rpc/handler.go ((*handler).handleBatch):

• Include error responses in responseBytes
• For example, account for the serialized JSON size of resp.Error
• Enforce maxResponseSize consistently for both success and error responses

Conceptually:

if resp.Error != nil {
    b, _ := json.Marshal(resp.Error)
    responseBytes += len(b)
} else {
    responseBytes += len(resp.Result)
}

(Exact implementation details can be refined.)

---

Tests

Add a regression test in rpc/:

• TestBatchResponseSizeLimitCountsErrorResponses

• Verify that large error responses:

  • Contribute to batch size accounting
  • Trigger -32003
  • Cause subsequent batch items to return -32003

Run with:

go test ./rpc

[fjl]

The idea is good, but I don't like that jsonError.Data has to be serialized twice to get the size. We have to force serialization of that field earlier and store it as json.RawMessage, then we could compute the size of the error without encoding it.