Build stall when CAS reset

[chrisstaite-menlo]

I had everything stall during a full Chromium build, it did all come back to life after a minute or so, but the cause appears to be this:

[2023-09-07T07:59:41.326Z WARN  h2::proto::streams::recv] recv_reset; remotely-reset pending-accept streams reached limit (20)
[2023-09-07T07:59:41.326Z ERROR cas] Failed running service : hyper::Error(Http2, Error { kind: GoAway(b"", ENHANCE_YOUR_CALM, Library) })

I'm not sure why this happened and it's the first time I've seen it.

[chrisstaite-menlo]

Getting quite a lot of this error from workers:

[2023-09-07T08:33:14.560Z WARN  simple_scheduler] Internal error for worker 46cd9913-c4bc-41f9-b1d3-cb3c3fe52d95: Error { code: Internal, messages: ["Receiver went away before receiving EOF : Failed to write eof to fast store in fast_slow store update : --- : Writer was dropped before EOF was sent : Failed to receive data in filesystem store : While processing with temp file \"/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/3147cfb3677571c42217a593af517bd73cd66990c8fcd08c9453030000000000-37957\" : Could not upload data to store in upload_file_to_store : for \"/root/.cache/turbo-cache/work/621d69738280b8cb78d47e763a09fc5f8166dfd0567b3523ecaa8659147b735c/by/goma/obj/third_party/blink/renderer/bindings/modules/v8/v8/v8_payment_response.o.d\" : Error while uploading results"] }

[allada]

It looks like the CAS was under a lot of load and the scheduler tried to do some lookups but the CAS was unable to process more streams due to some other bottleneck and resulted in the queue filling up, then dropping new connections.

We can mitigate some of this with:
https://doc.servo.org/h2/server/struct.Builder.html#method.max_pending_accept_reset_streams

However, in this case having more than one CAS might be needed in order to deal with excessive loads.

hyperium/h2#668

[chrisstaite-menlo]

It was running at a load average of 12 and had 300% CPU usage. I was trying to avoid having multiple CAS instances because it becomes a pain to manage. I may have to migrate to one of the S3 style storage solutions and host Turbo-Cache to front it. It should all have been serving out of the memory store though, so it's surprising that it couldn't handle it.

[allada]

I wounder if it got caught up in a lock somewhere? It would be great if this happens again if we could get a core dump.

The only N^2 problem that I am aware of in this project is on the scheduler matching, and since this happened on the CAS i'm certain it's a external influenced thing.

Do you happen to know the disk usage? Tokio's disk commands are not actually async, it actually uses blocking calls in a "block safe thread". By default the system is limited to a fixed amount of these blocking threads. So maybe it ended up with too much backpressure from the disk io?

[chrisstaite-menlo]

Didn't appear to have a particularly high wa but that's anecdotal.

[chrisstaite-menlo]

I'm actually getting a lot of this even when it's not heavily loaded:

[2023-09-07T14:06:14.316Z WARN  simple_scheduler] Internal error for worker 46cd9913-c4bc-41f9-b1d3-cb3c3fe52d95: Error { code: Internal, messages: ["Receiver went away before receiving EOF : Failed to write eof to fast store in fast_slow store update : --- : Writer was dropped before EOF was sent : Failed to receive data in filesystem store : While processing with temp file \"/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/06d463ed3e6600423456cdd4faf3c376f042ec06e38b9f002f2f040000000000-10824\" : Could not upload data to store in upload_file_to_store : Error while uploading results"] }
[2023-09-07T14:07:06.437Z WARN  simple_scheduler] Internal error for worker 46cd9913-c4bc-41f9-b1d3-cb3c3fe52d95: Error { code: Internal, messages: ["Receiver went away before receiving EOF : Failed to write eof to fast store in fast_slow store update : --- : Writer was dropped before EOF was sent : Failed to receive data in filesystem store : While processing with temp file \"/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/bd47cdb64c9fcf99f9b92f91ac9e407be88320381cc73a867d60040000000000-9453\" : Could not upload data to store in upload_file_to_store : Error while uploading results"] }
[2023-09-07T14:07:07.557Z WARN  simple_scheduler] Internal error for worker 46cd9913-c4bc-41f9-b1d3-cb3c3fe52d95: Error { code: Internal, messages: ["Receiver went away before receiving EOF : Failed to write eof to fast store in fast_slow store update : --- : Writer was dropped before EOF was sent : Failed to receive data in filesystem store : While processing with temp file \"/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/bdf50ad4b2d7bf49796d02e6f68de97c8c051c134cbb95ab5461040000000000-9838\" : Could not upload data to store in upload_file_to_store : Error while uploading results"] }
[2023-09-07T14:07:09.229Z WARN  simple_scheduler] Internal error for worker 46cd9913-c4bc-41f9-b1d3-cb3c3fe52d95: Error { code: Internal, messages: ["Receiver went away before receiving EOF : Failed to write eof to fast store in fast_slow store update : --- : Writer was dropped before EOF was sent : Failed to receive data in filesystem store : While processing with temp file \"/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/e7856deacd8906918a34ef8af8d16a120ba3d45fc678d8f2b262040000000000-9958\" : Could not upload data to store in upload_file_to_store : Error while uploading results"] }
[2023-09-07T14:07:14.642Z WARN  simple_scheduler] Internal error for worker 46cd9913-c4bc-41f9-b1d3-cb3c3fe52d95: Error { code: Internal, messages: ["Receiver went away before receiving EOF : Failed to write eof to fast store in fast_slow store update : --- : Writer was dropped before EOF was sent : Failed to receive data in filesystem store : While processing with temp file \"/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/0b4fff3fa0cd03d3e65189ba9ee94986feb26539ba1fcc3cfd6a040000000000-9197\" : Could not upload data to store in upload_file_to_store : Error while uploading results"] }
[2023-09-07T14:07:17.001Z WARN  simple_scheduler] Internal error for worker 46cd9913-c4bc-41f9-b1d3-cb3c3fe52d95: Error { code: Internal, messages: ["Receiver went away before receiving EOF : Failed to write eof to fast store in fast_slow store update : --- : Writer was dropped before EOF was sent : Failed to receive data in filesystem store : While processing with temp file \"/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/a350640a5de6855a692c7e81fd9b66dd1e2a27dffcccfd44ba70040000000000-12239\" : Could not upload data to store in upload_file_to_store : Error while uploading results"] }

I think we may have introduced a problem recently. I just updated the cluster to the latest main.

Only thing in the CAS logs are:

[2023-09-07T14:02:52.560Z ERROR bytestream_server] Write Resp: 2.7540956 None Status { code: Internal, message: "Stream error at byte 8388608 : In ByteStreamServer::write()", source: None }
[2023-09-07T14:02:52.561Z ERROR bytestream_server] Write Resp: 8.560164 None Status { code: Internal, message: "Stream error at byte 25165824 : In ByteStreamServer::write()", source: None }
[2023-09-07T14:02:52.586Z ERROR bytestream_server] Write Resp: 9.943344 None Status { code: Internal, message: "Stream error at byte 29360128 : In ByteStreamServer::write()", source: None }
[2023-09-07T14:02:52.675Z ERROR bytestream_server] Write Resp: 0.034807287 None Status { code: Internal, message: "Stream error at byte 2097152 : In ByteStreamServer::write()", source: None }
[2023-09-07T14:02:52.692Z ERROR bytestream_server] Write Resp: 3.1737354 None Status { code: Internal, message: "Stream error at byte 12582912 : In ByteStreamServer::write()", source: None }
[2023-09-07T14:02:52.727Z ERROR bytestream_server] Write Resp: 0.07661839 None Status { code: Internal, message: "Stream error at byte 2097152 : In ByteStreamServer::write()", source: None }
[2023-09-07T14:02:52.734Z ERROR bytestream_server] Write Resp: 0.0853766 None Status { code: Internal, message: "Stream error at byte 2097152 : In ByteStreamServer::write()", source: None }
[2023-09-07T14:08:35.537Z ERROR bytestream_server] Write Resp: 0.012445173 None Status { code: Internal, message: "Stream error at byte 20480 : In ByteStreamServer::write()", source: None }
[2023-09-07T14:08:35.559Z ERROR bytestream_server] Write Resp: 0.010165742 None Status { code: Internal, message: "Stream error at byte 24576 : In ByteStreamServer::write()", source: None }
[2023-09-07T14:08:40.401Z ERROR bytestream_server] Write Resp: 0.012167023 None Status { code: Internal, message: "Stream error at byte 65536 : In ByteStreamServer::write()", source: None }

[allada]

This would be very useful to get some metrics published under prometheus.

[chrisstaite-menlo]

I'm getting this on the worker:

[2023-09-07T08:32:24.893Z ERROR filesystem_store] Error { code: NotFound, messages: ["No such file or directory (os error 2)", "Failed to touch file in filesystem store \"/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934c945f030000000000-0\""] }
[2023-09-07T14:08:35.580Z ERROR filesystem_store] Error { code: NotFound, messages: ["No such file or directory (os error 2)", "Failed to touch file in filesystem store \"/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934c7ca3040000000000-0\""] }
[2023-09-07T14:08:35.618Z ERROR filesystem_store] Error { code: NotFound, messages: ["No such file or directory (os error 2)", "Failed to touch file in filesystem store \"/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934c7ca3040000000000-0\""] }
[2023-09-07T14:08:35.645Z ERROR filesystem_store] Error { code: NotFound, messages: ["No such file or directory (os error 2)", "Failed to touch file in filesystem store \"/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934c7ca3040000000000-0\""] }
[2023-09-07T14:08:35.645Z ERROR filesystem_store] Error { code: NotFound, messages: ["No such file or directory (os error 2)", "Failed to touch file in filesystem store \"/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934c7ca3040000000000-0\""] }
[2023-09-07T14:08:35.655Z ERROR filesystem_store] Error { code: NotFound, messages: ["No such file or directory (os error 2)", "Failed to touch file in filesystem store \"/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934c7ca3040000000000-0\""] }
[2023-09-07T14:08:35.655Z ERROR filesystem_store] Error { code: NotFound, messages: ["No such file or directory (os error 2)", "Failed to touch file in filesystem store \"/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934c7ca3040000000000-0\""] }
[2023-09-07T14:08:35.675Z ERROR filesystem_store] Error { code: NotFound, messages: ["No such file or directory (os error 2)", "Failed to touch file in filesystem store \"/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934c7ca3040000000000-0\""] }
[2023-09-07T14:08:35.676Z ERROR filesystem_store] Error { code: NotFound, messages: ["No such file or directory (os error 2)", "Failed to touch file in filesystem store \"/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934c7ca3040000000000-0\""] }
[2023-09-07T14:08:35.676Z ERROR filesystem_store] Error { code: NotFound, messages: ["No such file or directory (os error 2)", "Failed to touch file in filesystem store \"/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934c7ca3040000000000-0\""] }
[2023-09-07T14:08:35.676Z ERROR filesystem_store] Error { code: NotFound, messages: ["No such file or directory (os error 2)", "Failed to touch file in filesystem store \"/root/.cache/turbo-cache/data-worker-test/tmp_path-cas/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934c7ca3040000000000-0\""] }

[allada]

Can you try reverting 67b90e2c9a254687b7525053bb4153f95e216b9d and try?

That commit was significant and is the only major change I can think of that might cause this.

[chrisstaite-menlo]

Just doing a build with that reverted now.

[chrisstaite-menlo]

Now running with that build and it appears a lot happier, although it's only been up for a few seconds, so I'll try and exercise it to check.

[chrisstaite-menlo]

I've just hit it with a totally new branch and it's flying along without any issues. The CAS is running at a load average of 18 and rising and 300% CPU with a 2.8 wa but there's no stalls or breakages.

Looks like there might be an issue with 67b90e2.

[allada]

Dang, I will revert that commit. I may need to find another way to do what it's trying to do. I tried a few strategies, but this was the one i hated the least.

[chrisstaite-menlo]

I spoke too soon, getting stalls again. When it stalls the scheduler and remoteexec_proxy CPU usage drops to a tiny amount and get these:

[2023-09-07T14:54:23.990Z WARN  simple_scheduler] Internal error for worker 46ecd873-ab4f-4683-958f-c9b5b1db72b1: Error { code: Internal, messages: ["Receiver went away before receiving EOF : Failed to write eof to fast store in fast_slow store update : --- : Writer was dropped before EOF was sent : Failed to receive data in filesystem store : While processing with temp file /root/.cache/turbo-cache/data-worker-test/tmp_path-cas/fcbb86c63a3e9f318ef3e082d7cb809dcc9df658dd9b22a71477000000000000-10778 : Could not upload data to store in upload_to_store : Error while uploading results"] }
[2023-09-07T14:54:25.290Z WARN  simple_scheduler] Internal error for worker 46ecd873-ab4f-4683-958f-c9b5b1db72b1: Error { code: Internal, messages: ["Receiver went away before receiving EOF : Failed to write eof to fast store in fast_slow store update : --- : Writer was dropped before EOF was sent : Failed to receive data in filesystem store : While processing with temp file /root/.cache/turbo-cache/data-worker-test/tmp_path-cas/fe55b3977c64b4d0408e121f3e62406afb2258b73fbe6e192277000000000000-10888 : Could not upload data to store in upload_to_store : Error while uploading results"] }
[2023-09-07T14:54:27.344Z WARN  simple_scheduler] Internal error for worker dbfeef84-d1ad-4f6f-9f1e-9681195f2e35: Error { code: Internal, messages: ["Receiver went away before receiving EOF : Failed to write eof to fast store in fast_slow store update : --- : Writer was dropped before EOF was sent : Failed to receive data in filesystem store : While processing with temp file /root/.cache/turbo-cache/data-worker-test/tmp_path-cas/0d1a3dcffaedfe4a96e4ab48a6b97b627e407045c8393d63a378000000000000-8344 : Could not upload data to store in upload_to_store : for \"/root/.cache/turbo-cache/work/7403455e0ecaa6f782afa837ca22e09c11802d6424b144758d4dbec5de1d36f5/by/goma/obj/components/safety_check/safety_check/update_check_helper.o\" : Error while uploading results"] }
[2023-09-07T14:54:36.713Z WARN  simple_scheduler] Internal error for worker 46ecd873-ab4f-4683-958f-c9b5b1db72b1: Error { code: Internal, messages: ["Receiver went away before receiving EOF : Failed to write eof to fast store in fast_slow store update : --- : Writer was dropped before EOF was sent : Failed to receive data in filesystem store : While processing with temp file /root/.cache/turbo-cache/data-worker-test/tmp_path-cas/faef7fb3995348a490c099684ef8ff48c5a4d21a44cbaaf55079000000000000-8988 : Could not upload data to store in upload_to_store : for \"/root/.cache/turbo-cache/work/a9ef3a5608d364d407ed72a4542393a99aba5bdf327764eca13b7ba4a7f861ad/by/goma/obj/components/security_interstitials/core/core/bad_clock_ui.o.d\" : Error while uploading results"] }
[2023-09-07T14:54:37.671Z WARN  simple_scheduler] Internal error for worker 02ca6e82-c420-4286-b294-cb61181d3b9b: Error { code: Internal, messages: ["Receiver went away before receiving EOF : Failed to write eof to fast store in fast_slow store update : --- : Writer was dropped before EOF was sent : Failed to receive data in filesystem store : While processing with temp file /root/.cache/turbo-cache/data-worker-test/tmp_path-cas/6e6561700bbfbd48a75972966e5bf527a0f7cea14f3ff50cdf77000000000000-9735 : Could not upload data to store in upload_to_store : for \"/root/.cache/turbo-cache/work/ac584fe072761ea0a2a91e8f5a0b3a203e9db4880927581c3ce4cf405ef0d7ab/by/goma/obj/components/security_interstitials/core/core/safe_browsing_quiet_error_ui.o.d\" : Error while uploading results"] }
[2023-09-07T14:54:37.976Z WARN  simple_scheduler] Internal error for worker 02ca6e82-c420-4286-b294-cb61181d3b9b: Error { code: Internal, messages: ["Receiver went away before receiving EOF : Failed to write eof to fast store in fast_slow store update : --- : Writer was dropped before EOF was sent : Failed to receive data in filesystem store : While processing with temp file /root/.cache/turbo-cache/data-worker-test/tmp_path-cas/64f350e0acf2449906c1e7dea239a9fb5600e73b8a88f2e2fa77000000000000-8368 : Could not upload data to store in upload_to_store : for \"/root/.cache/turbo-cache/work/428835674a916f53f6e8b1fdde29879a719d8a6bb3216f5c954e9a57145861b6/by/goma/obj/components/security_interstitials/core/core/controller_client.o\" : Error while uploading results"] }
[2023-09-07T14:54:38.450Z WARN  simple_scheduler] Internal error for worker 02ca6e82-c420-4286-b294-cb61181d3b9b: Error { code: Internal, messages: ["Receiver went away before receiving EOF : Failed to write eof to fast store in fast_slow store update : --- : Writer was dropped before EOF was sent : Failed to receive data in filesystem store : While processing with temp file /root/.cache/turbo-cache/data-worker-test/tmp_path-cas/148722dcd9a71441c5783525482540187c087ae9d17246f21278000000000000-10085 : Could not upload data to store in upload_to_store : for \"/root/.cache/turbo-cache/work/6f35c0b9223d00b71422bec7c5a8a1a585f9bcd44a840064b25871c762071605/by/goma/obj/components/security_interstitials/core/core/https_only_mode_enforcelist.o.d\" : Error while uploading results"] }
[2023-09-07T14:54:38.509Z WARN  simple_scheduler] Internal error for worker dbfeef84-d1ad-4f6f-9f1e-9681195f2e35: Error { code: Internal, messages: ["Receiver went away before receiving EOF : Failed to write eof to fast store in fast_slow store update : --- : Writer was dropped before EOF was sent : Failed to receive data in filesystem store : While processing with temp file /root/.cache/turbo-cache/data-worker-test/tmp_path-cas/346bbdc66eeff2390a88ac5a1707ef7195645ef122d9c366bc7a000000000000-11609 : Could not upload data to store in upload_to_store : for \"/root/.cache/turbo-cache/work/a03735e7d1e744370ec03f9c39ad99fa6390ce6d607efca2d68a7fa31682cee7/by/goma/obj/components/segmentation_platform/internal/internal/field_trial_recorder.o.d\" : Error while uploading results"] }
[2023-09-07T14:54:39.107Z WARN  simple_scheduler] Internal error for worker 02ca6e82-c420-4286-b294-cb61181d3b9b: Error { code: Internal, messages: ["Receiver went away before receiving EOF : Failed to write eof to fast store in fast_slow store update : --- : Writer was dropped before EOF was sent : Failed to receive data in filesystem store : While processing with temp file /root/.cache/turbo-cache/data-worker-test/tmp_path-cas/9c060331fbf2cf65e497a425ff6f33bde624f64b8fbe41d84178000000000000-11684 : Could not upload data to store in upload_to_store : for \"/root/.cache/turbo-cache/work/5cff60d775f277c693c2ad6a7b4e1f39207dcd97ed67beb3a2adb273315382d9/by/goma/obj/components/segmentation_platform/internal/internal/metadata_utils.o.d\" : Error while uploading results"] }

[chrisstaite-menlo]

I think before I got these issues I was running a really old version back at around 259224b