CI should run st2-self-check for every PR

[arm4b]

Apart from Unit and Integration tests, we should also run the st2-self-check script for every PR as part of the CI pipeline.
Something we do in e2e and manual testing.

It should be possible with the Github Actions and would help us to catch bugs before merging into master.
Ex #5489.

[winem]

I am currently working on it. Should be ready for christmas. :)

[winem]

Still working on it but I have more challenges than expected with the tests in the GHA. So stay tuned. :)

[winem]

So I made some good progress on this topic I think.

I applied a bunch of changes mainly to the environment handling and to check if the scripts are executed in the context of GITHUB actions.

Now, all tests pass except for the remote and http actions (4 in total).

Honestly, I could not find a solution to pass these, yet.

So I have 2 questions:

1. does anyone have an idea how a solution where these tests pass as well may work?
2. would it be ok to remove them from the scope of the self-check if it's executed in Github Actions? So we could skip them if they are running at Github.

I think these are the last two questions to sort out. :)

The code can be found here: https://github.com/StackStorm/st2/tree/5496-self-check-in-gh-actions
An example CI run with the failing tests: https://github.com/StackStorm/st2/runs/5795166449?check_suite_focus=true

Any input is welcome!

[cognifloyd]

Remote tests are going to be an interesting solution to deal with. It would probably involve having it ssh into a docker container with sshd running...

But, I think we should disable those tests for now under GHA and move forward with the rest.

[arm4b]

@winem Could you open a draft PR for the https://github.com/StackStorm/st2/tree/5496-self-check-in-gh-actions ?
that would be easier for the team to find it, including the code diff, CI status, and progress.

BTW we have containers with sshd running. They're used for e2e package testing in st2-packages.

Could be a good fit for this case:
See https://github.com/StackStorm/st2packaging-dockerfiles/tree/master/packagingtest

[winem]

@cognifloyd I agree and it would be fine for me, too. We could handle the remaining tests in a dedicated PR.

But I'll take a look at @armab s link as well and see if we can re-use it here. A PR was created and linked.