HBASE-28898. Use reflection to access recoverLease(), setSafeMode() APIs. #6342

jojochuang · 2024-10-04T03:07:55Z

use reflection to access these Hadoop APIs.
added more UT coverage. Tested a few UTs with Hadoop 3.3.5.

TODO:

Test with Hadoop 3.3.6, 3.4.0.
Test with Ozone 2.0.0-SNAPSHOT.

Change-Id: I1a7b7bad0e8967553490b70439b18f8ddfed6d14

Change-Id: Ibf0ded6e104e9ed16f887f8411f4816eab8cea00

Change-Id: I351226ea55545dea309c7ed33c78c338cb42c07e

jojochuang · 2024-10-04T17:30:10Z

javac warning is unrelated.
[WARNING] /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-6342/yetus-general-check/src/hbase-asyncfs/src/test/java/org/apache/hadoop/hbase/util/TestRecoverLeaseFSUtils.java:[74,13] [NarrowCalculation] This product of integers could overflow before being implicitly cast to a long.

hbase-asyncfs/src/main/java/org/apache/hadoop/hbase/util/RecoverLeaseFSUtils.java

(1) Use FileSystem class instead of Object for file system object. (2) Added one more test case to properly test reflection code. Change-Id: I0710ae1023d81f68fb18ec3f8dd5b228d90df21e

Change-Id: I5e29bacec957d76555a0ff7483a462b614a8f3f5

Change-Id: Ie0b74f4801de33e0683e97e234cad9fcf13e985a

Change-Id: I4dd951945920220678677acefd27710eaf0f5658

stoty · 2024-10-16T08:39:20Z

Looks good to me.
Please fix the spotless and other warnings (unless unrelated).

I am a bit worried about the runtime errors.
I have not dug through the API, but I assume those errors can never happen with a released Hadoop version, right ?

jojochuang · 2024-10-16T16:13:00Z

The unit test failures do not reproduce in my local tree, so assuming flaky. But I wonder if the this change makes them easier to fail. Will dig into it.

Change-Id: Iec1149e25f6a8ac3f5fe4d2eb7287af9c0c75021

jojochuang · 2024-10-16T18:06:42Z

The spotbugs warning is unrelated. Will open another jira to track it.

stoty

Sorry, forgot to send my pending comments.
Adding them now.

stoty · 2024-10-16T08:28:34Z

hbase-asyncfs/src/main/java/org/apache/hadoop/hbase/util/RecoverLeaseFSUtils.java

+  private static Method recoverLeaseMethod = null;
+  public static final String LEASE_RECOVERABLE_CLASS_NAME = "org.apache.hadoop.fs.LeaseRecoverable";
+  static {
+    LOG.debug("RecoverLeaseFSUtils loaded");


nit: This should be after initializeRecoverLeaseMethod

Having this log message before the actual initialization is misleading.
Is there a specific reason for this ordering ?

Alternatively change the message to "initializing RecoverLeaseFSUtils"

stoty · 2024-10-16T08:30:29Z

hbase-asyncfs/src/main/java/org/apache/hadoop/hbase/util/RecoverLeaseFSUtils.java

+      LOG.debug(
+        "LeaseRecoverable interface not in the classpath, this means Hadoop 3.3.5 or below.");
+      try {
+        recoverLeaseMethod = DistributedFileSystem.class.getMethod("recoverLease", Path.class);


We expect to have this on all supported releases, right ?

Maybe log an error message to explain that this should never happen ?

stoty · 2024-10-16T08:34:17Z

hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java

+          .invoke(dfs, safeModeGet, true);
+        return (Boolean) ret;
+      } catch (InvocationTargetException | IllegalAccessException | NoSuchMethodException e) {
+        throw new RuntimeException(e);


There is no Hadoop release where we expect to get here, right ?

Maybe log an error message to explain that this should never happen ?

stoty · 2024-10-16T08:42:04Z

hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java

+    if (isDistributedFileSystem(dfs)) {
+      return ((DistributedFileSystem) dfs).setSafeMode(SAFEMODE_GET, true);
+    } else {
+      try {


Should we just return false here if we could not find the method previously ?
i.e. if this is an older Hadoop, and this is called with say, LocalFilesystem ?

Before calling isInSafeMode() it calls supportSafeMode(). If it's LocalFileSystem it will not enter isInSafeMode().

Change-Id: Iee3f259a46b5f12ed5948c61738a19ef77e29b5a

jojochuang · 2024-10-16T23:58:06Z

https://issues.apache.org/jira/browse/HBASE-28918 for the spotbug error

stoty

+1 LGTM

I have kicked off a new CI run

stoty

The log message in the static initializer is still misleading.

stoty · 2024-10-17T06:34:44Z

Please also re-run spotless:apply, the new log message does not comply with the spotless rules.

Change-Id: Iee20335148f41592ed5b36c7c526a544fce6ed87

jojochuang · 2024-10-19T00:31:27Z

Thanks for the review! Updated.

Apache-HBase · 2024-10-19T01:13:58Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 15s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ master Compile Tests _
+0 🆗	mvndep	0m 15s		Maven dependency ordering for branch
+1 💚	mvninstall	3m 0s		master passed
+1 💚	compile	3m 22s		master passed
+1 💚	checkstyle	0m 47s		master passed
-1 ❌	spotbugs	1m 34s	/branch-spotbugs-hbase-server-warnings.html	hbase-server in master has 1 extant spotbugs warnings.
+1 💚	spotless	0m 44s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 11s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 56s		the patch passed
+1 💚	compile	3m 23s		the patch passed
+1 💚	javac	3m 23s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 9s		The patch passed checkstyle in hbase-asyncfs
+1 💚	checkstyle	0m 37s		hbase-server: The patch generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11)
+1 💚	spotbugs	2m 15s		the patch passed
+1 💚	hadoopcheck	10m 25s		Patch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 💚	spotless	0m 44s		patch has no errors when running spotless:check.
			_ Other Tests _
+1 💚	asflicense	0m 20s		The patch does not generate ASF License warnings.
		38m 11s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6342/8/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#6342
Optional Tests	dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
uname	Linux 3b1f0123e8d9 5.4.0-192-generic #212-Ubuntu SMP Fri Jul 5 09:47:39 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `07c0831`
Default Java	Eclipse Adoptium-17.0.11+9
Max. process+thread count	84 (vs. ulimit of 30000)
modules	C: hbase-asyncfs hbase-server U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6342/8/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-10-19T04:33:52Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 30s		Docker mode activated.
-0 ⚠️	yetus	0m 2s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ master Compile Tests _
+0 🆗	mvndep	0m 21s		Maven dependency ordering for branch
+1 💚	mvninstall	2m 59s		master passed
+1 💚	compile	1m 11s		master passed
+1 💚	javadoc	0m 38s		master passed
+1 💚	shadedjars	5m 34s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 12s		Maven dependency ordering for patch
+1 💚	mvninstall	3m 0s		the patch passed
+1 💚	compile	1m 12s		the patch passed
+1 💚	javac	1m 12s		the patch passed
+1 💚	javadoc	0m 38s		the patch passed
+1 💚	shadedjars	5m 34s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	1m 0s		hbase-asyncfs in the patch passed.
+1 💚	unit	210m 28s		hbase-server in the patch passed.
		238m 4s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6342/8/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#6342
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux 16057e79f534 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `07c0831`
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6342/8/testReport/
Max. process+thread count	4902 (vs. ulimit of 30000)
modules	C: hbase-asyncfs hbase-server U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6342/8/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

stoty

+1 LGTM

jojochuang · 2024-10-23T20:05:04Z

Merged. Thanks @stoty

…PIs. (apache#6342) (cherry picked from commit 11c0742)

…PIs. (apache#6342) (cherry picked from commit 11c0742) (cherry picked from commit f373405)

…PIs. (apache#6342) (cherry picked from commit 11c0742) (cherry picked from commit f373405) (cherry picked from commit d2f0547)

…PIs. (#6342) (#6392) (cherry picked from commit 11c0742) (cherry picked from commit f373405) (cherry picked from commit d2f0547)

…PIs. (#6342) (#6391) (cherry picked from commit 11c0742)

…PIs. (#6342) (#6390) (cherry picked from commit 11c0742) (cherry picked from commit f373405)

Use reflection to access recoverlease(), setSafeMode() APIs.

49f4fa7

Change-Id: I1a7b7bad0e8967553490b70439b18f8ddfed6d14