HDDS-8101. Add FSO repair tool to ozone CLI in read-only and repair modes.

[DaveTeng0]

What changes were proposed in this pull request?

Bugs like HDDS-7592 can break the FSO tree and cause data to be orphaned in the OM. We have developed a tool to identify and repair this condition in the OM and tested it on affected clusters. This jira is to contribute the tool back to the community under the ozone CLI.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-8101

How was this patch tested?

Unit test, integration test.

[DaveTeng0]

cc. @errose28

[adoroszlai]

Thanks @DaveTeng0 for the patch.

Some comments about POM and CLI. Note: I haven't checked the code of the tool itself (FSORepairTool).

[errose28]

I think we still need to decide what the CLI for this should look like. We could do ozone {debug,repair} fso-tree or ozone repair fso-tree [--dry-run]. Also as we add more of these type of commands I think ones that are specific to a component should be under their own subcommand for organization, like ozone repair om fso-tree.

Attila also brought up the --dry-run mode. I think if the command is under repair only, then dry run would not be the expected default value. If we add the read-only invocation under debug then that becomes the equivalent of dry run and no flag is needed.

[DaveTeng0]

I think we still need to decide what the CLI for this should look like. We could do ozone {debug,repair} fso-tree or ozone repair fso-tree [--dry-run]. Also as we add more of these type of commands I think ones that are specific to a component should be under their own subcommand for organization, like ozone repair om fso-tree.

Attila also brought up the --dry-run mode. I think if the command is under repair only, then dry run would not be the expected default value. If we add the read-only invocation under debug then that becomes the equivalent of dry run and no flag is needed.

Yeah! extracted common codes between FSODebugCLI and FSORepairCLI to separated base classes FSOBaseCLI and FSOBaseTool, and make them reuse same logic.

[DaveTeng0]

Hello team! please feel free to let me know if there is any new comment~ Thanks!

[errose28]

Hi @DaveTeng0 I just did a quick pass on the code outside of the main repair logic.

There seems to be a lot of failures on your branch, although I haven't looked deeply and it could just be a bad CI run. Can you get that to a green state?

For the CLI, If we go the route of different debug and repair commands, I think we want ozone debug om fso-tree and ozone repair om fso-tree to be the two options. Just having ozone repair om fso-tree [--dry-run] isn't a bad approach either. I think I slightly prefer that one and it prevents the need for an extra om subcommand under each option (for now at least), but I'm ok with two separate commands as well.

[errose28]

@sarvekshayr will be resuming the work on this task. The remaining items to be addressed are:

• Print all messages to stdout. Remove log4j usage. A --verbose flag should indicate whether or not individual keys are printed.
• Check that RocksDB wrappers are being used correctly and no banned imports are failing the build.
  • Direct usage of RocksDB will fail the build.
• Add --bucket and --volume CLI flags to allow the tool to only focus on problematic areas.
• Add disclaimer to the output that the tool is currently not supported with snapshots.
• Estimate replicated size if possible.
• Create a follow up Jira to support snapshots

[adoroszlai]

Continued in #7368.