Use regex-lint to provide circular dependency checking in pants

[cognifloyd]

Background

This is another part of introducing pants, as discussed in the TSC Meetings on 12 July 2022, 02 Aug 2022, 06 Sept 2022, and 04 Oct 2022. Pants has fine-grained per-file caching of results for lint, fmt (like black), test, etc. It also has lockfiles that work well for monorepos that have multiple python packages. With these lockfiles CI should not break when any of our dependencies or our transitive dependencies release new versions, because CI will continue to use the locked version until we explicitly relock with updates.

To keep PRs as manageable/reviewable as possible, introducing pants will take a series of PRs. I do not know yet how many PRs; I will break this up into logical steps with these goals:

• introduce pants to the st2 repo, and
• teach some (but not all) TSC members about pants step-by-step.

Other pants PRs include:

• Initialize pantsbuild #5713
• Configure pantsbuild source roots #5724
• Enable pantsbuild language backends #5725
• Adjust .gitignore to work better with pantsbuild #5726
• Add GHA workflow to validate pants BUILD files #5732
• Configure pants_ignore and bump pants to v2.14.0rc1 #5733
• Explicitly disable pantsbuild opt-in anonymous telemetry #5737
• Create pantsbuild BUILD files with ./pants tailor :: #5738
• Add GHA workflow to lint with pantsbuild #5758
• Use shellcheck (via pantsbuild) #5751
• Use black (via pantsbuild) #5774
• Use flake8 (via pantsbuild) #5776
• Use bandit (via pantsbuild) #5777

Overview of this PR

The Makefile has some checks for circular dependencies. Pants has a builtin regex-lint subsystem that lets us do basically the same thing via pants.

These are the relevant targets in the Makefile:

st2/Makefile

Lines 578 to 591 in 94d0798

	.PHONY: .st2client-dependencies-check
	.st2client-dependencies-check:
	@echo "Checking for st2common imports inside st2client"
	find ${ROOT_DIR}/st2client/st2client/ -name \*.py -type f -print0 | xargs -0 cat | grep st2common ; test $$? -eq 1
	.PHONY: .st2common-circular-dependencies-check
	.st2common-circular-dependencies-check:
	@echo "Checking st2common for circular dependencies"
	find ${ROOT_DIR}/st2common/st2common/ -name \*.py -type f -print0 | xargs -0 cat | grep st2reactor ; test $$? -eq 1
	find ${ROOT_DIR}/st2common/st2common/ \( -name \*.py ! -name runnersregistrar\.py -name \*.py ! -name compat\.py ! -name inquiry\.py \) -type f -print0 | xargs -0 cat | grep st2actions ; test $$? -eq 1
	find ${ROOT_DIR}/st2common/st2common/ -name \*.py -type f -print0 | xargs -0 cat | grep st2api ; test $$? -eq 1
	find ${ROOT_DIR}/st2common/st2common/ -name \*.py -type f -print0 | xargs -0 cat | grep st2auth ; test $$? -eq 1
	find ${ROOT_DIR}/st2common/st2common/ \( -name \*.py ! -name router\.py -name \*.py \) -type f -print0 | xargs -0 cat | grep st2stream; test $$? -eq 1
	find ${ROOT_DIR}/st2common/st2common/ -name \*.py -type f -print0 | xargs -0 cat | grep st2exporter; test $$? -eq 1

It turns out that the Makefile targets are somewhat out-of-date: For example, some of the listed exceptions to the rules no longer apply and we removed st2exporter in #5676. I documented each of my updates in comments of the lint config file.

In #5776, I had to add our st2flake8 plugin for flake8 which handles checking for the copyright. We can replace that plugin with regex-lint. I noted how we can do that in the regex-lint config file, but I left it commented out for now. We can enable it later when we're ready to retire the st2flake8 custom plugin, which we can probably do once we're ready to remove the Makefile and related files.

Relevant Pants documentation

• ./pants generate-lockfiles goal
• ./pants lint goal
• regex-lint backend\

Things you can do with pantsbuild

You can run ./pants lint :: to see if regex-lint finds any issues (the GHA Lint workflow runs this as well). You can run only one of the linters with the helpful --only arg like this:

$ ./pants lint --only=regex-lint ::
13:21:52.61 [INFO] Initializing scheduler...
13:21:52.93 [INFO] Scheduler initialized.
13:21:54.27 [INFO] Completed: Lint with regex patterns - regex-lint succeeded.
385 files matched all required patterns.
0 files failed to match at least one required pattern.



✓ regex-lint succeeded.

The --only flag is "scoped" to the lint goal. So, these are equivalent if you wanted to specify that flag in a different order:
./pants lint --only=regex-lint ::
./pants --lint-only=regex-lint lint ::
./pants lint :: --lint-only=regex-lint

note: that I had to add lint in the arg name to use a different order

[rush-skills]

LGTM

[rush-skills]

What does inverted mean here?

[cognifloyd]

Search for st2common but mark files as successful if they do NOT have the string.