idea: linting dependencies.yaml files via '--strict'

[jameslamb]

Description

I just manually updated a LOT of dependencies.yaml files across RAPIDS (rapidsai/build-planning#31), and found myself looking for some things manually that should be possible to enforce with a linter.

This issue proposes adding a --strict argument to rapids-dependency-file-generator, which enforces more standardization across dependencies.yaml files.

Benefits of this work

• greater standardization of dependencies.yaml files should reduce the effort to do all-of-RAPIDS migrations (either manually or with rapids-reviser)
• may help to reduce complexity of the files
• could save some reviewer effort (e.g. the need to leave comments that are about personal preference and style)

Acceptance Criteria

• rapids-dependency-file-generator enforces some linting checks on dependencies.yaml files
• that enforcement is opt-in (off by default)
• that enforcement is deployed across all RAPIDS repos .pre-commit-config.yaml configurations using rapids-dependency-file-generator as a hook

Approach

I'm proposing that if a flag --strict is passed to rapids-dependency-file-generator, it check the content of the YAML file passed to --config and raise a non-0 exit code if any of a set of opt-in linting rules are violated.

I'll list some initial ideas I have here. I'm sure others will have more. For most of these, I have lightly-held opinions about which should be the preferred pattern, and care more that there be some preferred pattern and a tool to automatically enforce that preference.

• preferring indented lists to inline [] lists

# this
packages:
  - rmm-cu12

# not this
packages: [rmm-cu12]

• preferring indented maps to inline {} maps

# this
matrix:
  cuda: "12.*"

# not this:
matrix: {"cuda": "12.*"}

• preferring explicit : null to implicit (which might catch some mistakes)

# this
- matrix: null
  packages: null

# not this
- matrix:
  packages:

• disallowing any anchors that are never re-used

# failing if this is never met by a corresponding '*cupy_cu12'
- matrix:
    cuda: "12.*"
   packages:
     - &cupy_cu12 cupy-cuda12x>=12.0.0

Notes

This proposal is inspired by mypy --strict (mypy docs).

And by conversations like this: rapidsai/rmm#1627 (comment)

[jameslamb]

Thinking more about this this morning.... all of the examples I gave are generic YAML-formatting things and not specific to rapids-dependency-file-generator. Maybe just yamllint or similar could be used to enforce those things if we wanted, with no changes to rapids-dependency-file-generator.