[Tooling] Flaky tests

[deblasis]

Objective

Tests should be deterministic

Origin Document

It looks like that some of our tests are flaky, just logging this for now, I need to dive deeper to understand the root cause(s), it might be something very easy.

A couple of examples:

https://github.com/pokt-network/pocket/runs/8227176834?check_suite_focus=true
https://github.com/pokt-network/pocket/runs/8187453124?check_suite_focus=true

But it can be verified locally with multiple executions of

make test_all

First run 👍

Second run 💩

Goals

• Identify the flaky tests
• Fix them
• Document (if you think it's necessary) the root cause for future reference

Deliverable

• Deterministic test suite

Non-goals / Non-deliverables

• ...

[Optional] Testing Methodology

• All tests: make test_all
• LocalNet: verify a LocalNet is still functioning correctly by following the instructions at docs/development/README.md

---

Creator: @deblasis
Co-Owners: @Olshansk

[Olshansk]

I've been meaning to fix this for a while but just haven't gotten around to it.

The core of the issue is that we're using a Pacemaker to timeout certain steps in the HotPOKT consensus pipeline, and I'm using timeouts to simulate quick rounds.

If you're willing to dive into the code and take a stab at this, I think that would be a great learning experience. Lmk if you'd want to do a "pair coding" walkthrough first!

[deblasis]

Say no more. Sounds like an unmissable opportunity.

I'll dive into the code and if after hitting a few "walls" with my head I start feeling like I am losing consciousness (metaphorically speaking 😆) I'll reach out for help :)

At a glance, from what you are saying, it seems that perhaps time should be mocked so that tests don't rely on waiting for actual time to pass.
I have dealt with similar race conditions when testing high-frequency trading algos in the past but in other languages, I think it was C#.

Anyway, I'll have a lookie and update here / cry for help via DM.

[Olshansk]

At a glance, from what you are saying, it seems that perhaps time should be mocked so that tests don't rely on waiting for actual time to pass.

That is correct. I was thinking that there is an opportunity to also simply use channels and block with a timeout, to avoid the race conditions, but haven't driven into it.

Agreed that it's a great idea, so let me know when/if you hit a wall.