Add enhanced CI to capture common doc problems

[karenzone]

Our docs tooling is a workhorse. But sometimes, it burps and introduces plugin regressions for a wide variety of reasons. We pass docs-ci, even though there are changes lurking in the PR that we don't want to merge. @jsvd suggested that I capture the heuristics I use to review plugin updates so that we can investigate adding additional CI to identify and notify on these problems.

Sample PRs with regressions: elastic/logstash-docs#1434

Indications that a regression has sneaked in:

• Plugin date and/or publish date should never go backwards, except:
  • if we have to yank a plugin (which rarely happens). Even if we yank a plugin version, I don't believe that docs go backwards. Regardless, that would be a special case that requires evaluation and possible intervention.
  • On initial docgens for a new branch cut from main. Why? Because main doesn't have a Gemfile.jruby-2.6.lock.release file. The main branch in logstash-docs might allow more recent plugins than are allowed for the newly cut branch.
• Docs content changes without an update to version/release date.
  • @mashhurs has made some wonderful fixes (Plugin doc generation: preserve content in logstash-docs if no version bump #45 and Plugin doc generation: preserve content in logstash-docs if no version bump (integrations) #79) to keep docgen from stomping manual fixes in logstash-docs under normal processing. If we see content with no version bump, that indicates that a situation outside of "normal processing" has occurred and needs to be investigated.

This falls under the category of linting, but adding to my wish list just in case:

• Asciidoc formatting requires a return after an introductory sentence and before the bulleted items. Markdown does not have this requirement. An improperly formatted list is quite common, resulting in the intro paragraphs and all bullet items being munged into one gigantic paragraph. Docs-ci doesn't catch this.

Note: Currently, the manual approval process also gives us a chance to find and fix problems in the changelog. Some changelog linting would be a good thing, too.

Expanding scope to include Doc quality

The previous suggestions are mostly related to catching regressions in plugin docs for the Logstash Reference (LSR). This section expands into the realm of general doc quaility.

• Plugin options should be listed in alphabetical order in both the options table and in the descriptions listing. New options show up as isolated from context in docs PRs, and peer reviewers often fail to check for this.

Related

Keep an eye on this effort, too: https://github.com/elastic/docs-lint/issues/37

Next steps

When we have this tooling in place and working reliably, we can investigate additional automation for plugin docs.

Deploy at the plugin repo level

This additional checking should be deployed at the plugin repo level to alert contributors to problems before they merge the PR, bump versions, publish the plugin. This approach would help prevent problems at the source where they can be resolved before they get "in the wild" where they require more work to fix.

[karenzone]

Another example:

• auto generated update of versioned plugin documentation (contains truncations) logstash-docs#1625
• Doc: Troubleshoot docs-ci failure related to kafka output docs logstash-plugins/logstash-integration-kafka#163

[karenzone]

Closing for now. New docs formats and tooling will require new strategies and approaches.