Generator tests are flaky

[maribethb]

[maribethb]

Additional info from debugging:

• It's hard to run a bisect for this since they are flaky, and mostly on CI. I have yet to see this test fail locally.
• Scrolling back through our Actions, feat: procedure blocks update models is the first one that fails after 6hr that I see that is probably related. There is an earlier one (https://github.com/google/blockly/actions/runs/3500833960/jobs/5863894016) but it's the mocha tests that fail there. There are also other tests that fail in between but they are all from Beka's procedure stuff and fail with different errors (in the ones I clicked on, the generator tests don't even seem to be running so not sure what's up there)
• I tried moving the test output to a different directory (temp/generators) and it didn't help
• I also tried reverting fix(tests): Fix bootstrapping of generators in compressed mode #6703 because, idk, seemed mildly related i guess and it was right before Merge master to develop #3606 in the GitHub Actions list but that also did not help

So I still do not know the root cause or even when this started happening and I'm not sure how else to debug

[maribethb]

Pretty sure I've root caused it. The scripts in webdriver are timing out.

In the generator test, we had a .catch(() => {}) statement which was silencing any errors from webdriver. If I change that to log the error that was caught, I saw "Script timeout". So I increased the timeout to 2 minutes (default is 30 seconds) and will try re-running that a few times to see if it's less flaky. If so I'll submit a PR to fix this tomorrow.

This would explain why it's flaky since sometimes it is fast enough to complete.

[cpcallen]

• I also tried reverting fix(tests): Fix bootstrapping of generators in compressed mode #6703 because, idk, seemed mildly related i guess…

I'm beginning to wonder if there's any point my submitting PRs, since you keep reverting them… ;-)

and it was right before #3606 in the GitHub Actions list but that also did not help

Actual question: did you mean something other than #3606?

In the generator test, we had a .catch(() => {}) statement which was silencing any errors from webdriver.

🙄 Good find. Sorry I didn't catch that when reviewing #6431!

In fact the test should fail if the call throws (as it should in a few other places too).

[maribethb]

Haha, sorry Christopher. I was grasping at straws in the list of PRs, until that failed and I tried good ol' print statement debugging. #6767 shall live on as a record of my thought process lol.

And yes, I meant #6672. If you click the first link in the quoted comment, you'll see that next to the PR name it says #3606 and I now have no idea what that refers to...

[cpcallen]

And yes, I meant #6672. If you click the first link in the quoted comment, you'll see that next to the PR name it says #3606 and I now have no idea what that refers to...

Oh yes: it seems that GitHub numbers github actions invocations (i.e., the job, which might get run several times if necessary) with a separate sequence from the one used for issues and PRs. (I was a little annoyed when I first discovered that issues and PRs shared a sequence, but now we see the consequences of having separate sequences without some unique prefix like b/XXX vs. cl/XXX… :-(