This document describes how to get up and running with Apache Yetus,
as pertains to the development of Apache HBase. Specifically this covers the use
of test-patch, of which HBase developers make use for tasks related to code
quality. These are normally run via automation in the foundation's
Build infrastructure. They can also be run locally, which is the
subject of this document.
The Yetus project provides its own documentation of test-patch in
Basic PreCommit. By comparison, this document is
intended to be highly abbreviated, hands-on, and focused on the HBase use-case.
See that document for more complete explanations and further details.
In order to run Yetus, you'll need to first install Yetus and its dependencies. This is somewhat simplified when used in Docker mode. Yetus can be retrieved from a distribution artifact. Homebrew/Linuxbrew users can install from the tap, the process for which is also described on the downloads page.
Apache Yetus is comprised of a number of different components. The focus of our
interest is test-patch. test-patch is a modular system. Many modules depend
on some external tool to provide the underlying functionality. For example, the
compile check delegates to a number of provider modules, for example maven
or gradle for JVM projects. In order to use these modules, those tools must
be installed. Yetus calls these modules "plugins".
To see a list of all plugins available to test-patch, use
$ test-patch --list-pluginsTo see a list of all the options available in both the core modules as well as all the plugins, use
$ test-patch --plugins=all --helpAn invocation of test-patch requires use of one or more plugins. Often times,
when the full suite of checks are run, specify the meta-plugin "all". Otherwise,
a limited selection of plugins can be selected using the --plugins argument.
test-patch operates within a "workspace," a checkout from a source control
repository. It has a number of options pertaining to this workspace, such as
the path to the workspace (--basedir) and whether it will permit the presence
of uncommitted changes therein (--dirty-workspace).
Onto this workspace, it can optionally apply a change, a.k.a., the "patch" in
"test patch." The patch can come from a number of sources, including a patch
file, a JIRA ID, a Pull Request, &c. Or, explicitly inform Yetus that no patch
file is provided, and the repository should be checked as is, by passing
--empty-patch.
test-patch is extremely extensible. Even the functionality of its core
modules can be extended or overridden. It allows for this type of
"personalization" by way of "personality" overrides. Yetus ships a number of
these personalities; a pre-packaged personality can be selected via the
--project parameter. There is a provided HBase personality in Yetus, however
the HBase project maintains its own within the HBase source repository. Specify
the path to the personality file using --personality. The HBase repository
places this file under dev-support/hbase-personality.sh.
Running Yetus in Docker mode simplifies the concerns of dependencies because
the provided Dockerfile handles their installation automatically. However,
for MacOS users, there are a number of known issues with running Docker on OSX,
so it may be preferable to instead run outside of the container.
To run in Docker, of course you must install docker or some container runtime
equivalent. Docker Desktop is an option for Mac and Windows
users.
When running test-patch with HBase in docker mode, two parameters must be
provided. --docker activates the docker module, enabling the feature.
Additionally, --dockerfile points Yetus at HBase's provided Dockerfile,
instead of using the one provided out-of-the-box by Yetus. The HBase repository
places this file under dev-support/docker/Dockerfile.
HBase supports compilation and test execution on a number of different JVM
versions. To facilitate this, the Dockerfile installs multiple JVMs. An
invocation of test-patch requires additional parameters in order to specify
the target JVM. How you do this is slightly different based on whether you run
with or without Docker mode.
Simply specify JAVA_HOME in the environment in which test-patch is launched.
The docker image JDKs are available under /usr/lib/jvm. Symlinks are provided
for each supported major version, i.e., java-11. Use the test-patch argument
--java-home.
One of the checks supported by Yetus is the static analysis tool
SpotBugs. Let's put together all the above and run SpotBugs with
test-patch.
The SpotBugs check depends on a SpotBugs installation. This is provided by the docker file. For running without docker, you'll need to download and unpack the SpotBugs binary distribution.
Running the SpotBugs check also depends on Apache Maven, the build
system used by HBase. In order for the check to function, both the maven
and spotbugs plugins must be specified. If this dependency is omitted, Yetus
will error with a message similar to
ERROR: you can't specify maven as the buildtool if you don't enable the plugin.
To run just the SpotBugs check, we must explicitly specify spotbugs and its
dependency using --plugins=maven,spotbugs.
Putting it all together, without using docker:
$ cd /path/to/hbase
$ JAVA_HOME=/path/to/jdk-8 test-patch.sh \
--plugins=maven,spotbugs \
--spotbugs-home=/path/to/spotbugs/
--dirty-workspace \
--empty-patch \
--personality=./dev-support/hbase-personality.shPutting it all together, using docker:
$ cd /path/to/hbase
$ test-patch.sh \
--plugins=maven,spotbugs \
--dirty-workspace \
--empty-patch \
--personality=./dev-support/hbase-personality.sh \
--docker \
--dockerfile=./dev-support/docker/Dockerfile \
--java-home=/usr/lib/jvm/java-8