Add pants-plugins/release to handle wheel metadata generation

[cognifloyd]

Background

This is another part of introducing pants, as discussed in various TSC meetings.

Related PRs can be found in:

• pantsbuild milestone (which includes PRs since Initialize pantsbuild #5713)
• pantsbuild label (which also includes preparatory PRs that came before adding pantsbuild)

Overview of this PR

Generally, wheels have a setup.py file in them. Right now, we have hard-coded setup.py and a copy of dist_utils.py. These are spread out, and no one likes maintaining them. But, using pants, this becomes much easier.

Pants can auto-generate a setup.py for us whenever it builds a python_distribution() (ie a wheel or an sdist). And, pants makes it easy to centralize all of the common bits of metadata that need to go in all the wheels (like author="StackStorm" or our project_urls). This PR adds pants-plugins/release to take advantage of those features.

python_distribution() targets

This PR does not actually create the python_distribution() targets yet--that will happen in a follow-up PR. But, here is what a python_distribution looks like with some common options we will use:

python_distribution(
    name="myrunner",
    dependencies=[...],
    provides=python_artifact(
        name="myrunner",
        version="1.2.3",
        ...
        scripts=[...],  # for our pre-made scripts
    ),
    entry_points={
        "st2common.runners.runner": {...},
        "console_scripts": {...},  # for auto-generated entry point scripts
    }
)

Pants generates setup.py

Everything in provides=python_artifact(...) becomes the keyword args that pants puts in the call to setup(...) in the generated setup.py file. setup(...) also gets kwargs from a few other places including the entry_points= kwarg of python_distribution, and a special plugin rule:

st2/pants-plugins/release/rules.py

Lines 88 to 89 in 2266086

	@rule
	async def setup_kwargs_plugin(request: StackStormSetupKwargsRequest) -> SetupKwargs:

NB: Only one plugin can provide the SetupKwargs.

To wire up this rule, I created a StackStormSetupKwargsRequest class, and wired it up with a UnionRule:

st2/pants-plugins/release/rules.py

Line 165 in 2266086

UnionRule(SetupKwargsRequest, StackStormSetupKwargsRequest),

That way, whenever pants goes to generate setup.py, it will use our plugins to inject additional kwargs in the setup(...) call of setup.py.

Dynamically retrieving version and long_description kwargs

In our current setup.py + dist_utils.py setup, we retrieve the version kwarg from an __init__.py file like this (or for st2client we actually just import __version__ from the __init__.py file):

st2/st2common/setup.py

Line 36 in 2266086

version=get_version_string(INIT_FILE),

The get_version_string function is defined in dist_utils.py:

st2/st2common/dist_utils.py

Lines 163 to 174 in 2266086

	def get_version_string(init_file):
	"""
	Read __version__ string for an init file.
	"""
	with open(init_file, "r") as fp:
	content = fp.read()
	version_match = re.search(r"^__version__ = ['\"]([^'\"]*)['\"]", content, re.M)
	if version_match:
	return version_match.group(1)
	raise RuntimeError("Unable to find version string in %s." % (init_file))

So, we do something very similar in this plugin. The BUILD file specifies a version_file arg on python_artifact(...), to say which __init__.py file to use. Then the plugin gets the contents of the file and extracts the version string, just like the get_version_string function from our old dist_utils.py file.

NB: The glob_match_error_behavior=GlobMatchErrorBehavior.error bit means that if the version_file does not exist, an error will be raised when attempting to build the wheel.

st2/pants-plugins/release/rules.py

Lines 100 to 108 in 2266086

	version_digest_contents, readme_digest_contents = await MultiGet(
	Get(
	DigestContents,
	PathGlobs(
	[f"{request.target.address.spec_path}/{version_file}"],
	description_of_origin=f"StackStorm version file: {version_file}",
	glob_match_error_behavior=GlobMatchErrorBehavior.error,
	),
	),

st2/pants-plugins/release/rules.py

Lines 118 to 125 in 2266086

	version_file_contents = version_digest_contents[0].content.decode()
	version_match = re.search(
	r"^__version__ = ['\"]([^'\"]*)['\"]", version_file_contents, re.M
	)
	if not version_match:
	raise ValueError(
	f"Could not find the __version__ in {request.target.address.spec_path}/{version_file}\n{version_file_contents}"
	)

In our current setup.py for the st2client wheel, we also include a README.rst file in the long_description kwarg. So, we do the same thing in this plugin as well:

st2/pants-plugins/release/rules.py

Lines 109 to 115 in 2266086

	Get(
	DigestContents,
	PathGlobs(
	[f"{request.target.address.spec_path}/README.rst"],
	glob_match_error_behavior=GlobMatchErrorBehavior.ignore,
	),
	),

st2/pants-plugins/release/rules.py

Lines 132 to 137 in 2266086

	long_description = (
	readme_digest_contents[0].content.decode() if readme_digest_contents else ""
	)
	if long_description:
	hardcoded_kwargs["long_description_content_type"] = "text/x-rst"
	hardcoded_kwargs["long_description"] = long_description

Centralized Wheel Metadata

And finally, we also define the metadata that is common to all of our wheels here:

st2/pants-plugins/release/rules.py

Lines 37 to 66 in 2266086

	PROJECT_METADATA = dict(
	author="StackStorm",
	author_email="[email protected]",
	url="https://stackstorm.com",
	license="Apache License, Version 2.0",
	# dynamically added:
	# - version (from version_file)
	# - long_description (from README.rst if present)
	# - long_description_content_type (text/x-rst)
	)
	PROJECT_URLS = {
	# TODO: use more standard slugs for these
	"Pack Exchange": "https://exchange.stackstorm.org",
	"Repository": "https://github.com/StackStorm/st2",
	"Documentation": "https://docs.stackstorm.com",
	"Community": "https://stackstorm.com/community-signup",
	"Questions": "https://github.com/StackStorm/st2/discussions",
	"Donate": "https://funding.communitybridge.org/projects/stackstorm",
	"News/Blog": "https://stackstorm.com/blog",
	"Security": "https://docs.stackstorm.com/latest/security.html",
	"Bug Reports": "https://github.com/StackStorm/st2/issues",
	}
	META_CLASSIFIERS = (
	"Development Status :: 5 - Production/Stable",
	"Intended Audience :: Information Technology",
	"Intended Audience :: Developers",
	"Intended Audience :: System Administrators",
	"License :: OSI Approved :: Apache Software License",
	)
	LINUX_CLASSIFIER = "Operating System :: POSIX :: Linux"

Note that the classifiers get combined like this:

st2/pants-plugins/release/rules.py

Lines 151 to 157 in 2266086

	kwargs["classifiers"] = (
	*META_CLASSIFIERS,
	LINUX_CLASSIFIER,
	# TODO: add these dynamically based on interpreter constraints
	*python_classifiers("3", "3.6", "3.8"),
	*kwargs.get("classifiers", []),
	)

That means that the python version classifiers are also included. As the TODO says, we still need to pull out the versions from the interpreter_constraints--until then, we've just got them hard-coded.

Developing pants plugins

Pants has extensive documentation on the plugin API, including how to create targets, how to write rules, and there's even a mini tutorial about how to add a formatter (adding formatters is a very common thing for plugins to do).

• Plugins overview
• Testing Plugins
• python_distribution() target
• Custom python_artifact() kwargs

[cognifloyd]

I also noticed that this URL in our old st2client/setup.py file was out-of date. Ideally we won't be using this setup.py file, but I went ahead and updated it here as well.

[cognifloyd]

Rebased and added pants-plugins/README.md entry.