HDFS-17358. EC: infinite lease recovery caused by the length of RWR equals to zero or datanode does not have the replica. by hfutatzhanghb · Pull Request #6509 · apache/hadoop

hfutatzhanghb · 2024-01-28T14:47:37Z

Description of PR

Recently, there is a strange case happened on our ec production cluster.

The phenomenon is as below described: NameNode does infinite recovery lease of some ec files(~80K+) and those files could never be closed.

After digging into logs and releated code, we found the root cause is below codes in method BlockRecoveryWorker$RecoveryTaskStriped#recover:

          // we met info.getNumBytes==0 here! 
          if (info != null &&
              info.getGenerationStamp() >= block.getGenerationStamp() &&
              info.getNumBytes() > 0) {
            final BlockRecord existing = syncBlocks.get(blockId);
            if (existing == null ||
                info.getNumBytes() > existing.rInfo.getNumBytes()) {
              // if we have >1 replicas for the same internal block, we
              // simply choose the one with larger length.
              // TODO: better usage of redundant replicas
              syncBlocks.put(blockId, new BlockRecord(id, proxyDN, info));
            }
          }

          // throw exception here!
          checkLocations(syncBlocks.size());

The related logs are as below:

java.io.IOException: BP-1157541496-10.104.10.198-1702548776421:blk_-9223372036808032688_2938828 has no enough internal blocks, unable to start recovery. Locations=[...]

2024-01-23 12:48:16,171 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: initReplicaRecovery: blk_-9223372036808032686_2938828, recoveryId=27615365, replica=ReplicaUnderRecovery, blk_-9223372036808032686_2938828, RUR getNumBytes() = 0 getBytesOnDisk() = 0 getVisibleLength()= -1 getVolume() = /data25/hadoop/hdfs/datanode getBlockURI() = file:/data25/hadoop/hdfs/datanode/current/BP-1157541496-x.x.x.x-1702548776421/current/rbw/blk_-9223372036808032686 recoveryId=27529675 original=ReplicaWaitingToBeRecovered, blk_-9223372036808032686_2938828, RWR getNumBytes() = 0 getBytesOnDisk() = 0 getVisibleLength()= -1 getVolume() = /data25/hadoop/hdfs/datanode getBlockURI() = file:/data25/hadoop/hdfs/datanode/current/BP-1157541496-10.104.10.198-1702548776421/current/rbw/blk_-9223372036808032686

because the length of RWR is zero, the length of the returned object in below codes is zero. We can't put it into syncBlocks.
So throw exception in checkLocations method.

ReplicaRecoveryInfo info = callInitReplicaRecovery(proxyDN,new RecoveringBlock(internalBlk, null, recoveryId));

hadoop-yetus · 2024-01-28T19:42:20Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	7m 55s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	32m 23s		trunk passed
+1 💚	compile	0m 49s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 45s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	checkstyle	0m 39s		trunk passed
+1 💚	mvnsite	0m 50s		trunk passed
+1 💚	javadoc	0m 44s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 9s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	1m 53s		trunk passed
+1 💚	shadedclient	21m 22s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 41s		the patch passed
+1 💚	compile	0m 45s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 45s		the patch passed
+1 💚	compile	0m 41s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	javac	0m 41s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 32s		the patch passed
+1 💚	mvnsite	0m 42s		the patch passed
+1 💚	javadoc	0m 34s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 2s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	1m 54s		the patch passed
+1 💚	shadedclient	21m 13s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	195m 42s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 27s		The patch does not generate ASF License warnings.
		293m 39s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestLeaseRecovery
	hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
	hadoop.hdfs.TestLeaseRecoveryStriped

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6509/1/artifact/out/Dockerfile
GITHUB PR	#6509
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux d1e5494161f3 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `30e3bda`
Default Java	Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6509/1/testReport/
Max. process+thread count	4210 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6509/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hfutatzhanghb · 2024-01-29T09:56:29Z

@Hexiaoqiao @zhangshuyan0 @tomscut Sir, could you please take a look at this PR when you have free time? Thanks a lot.

zhangshuyan0

Greate catch! I also reproduced this bug in my testing environment. If the client crashes when the amount of data written is less than 6 cells (use RS-6-3 for example), the file will never be able to be closed.
If you could add a UT, that would be even better.

hfutatzhanghb · 2024-01-29T10:59:38Z

Thanks a lot for responsing, Sir. Will add an UT soonly.

…

---- Replied Message ---- | From | ***@***.***> | | Date | 01/29/2024 18:45 | | To | ***@***.***> | | Cc | ***@***.***>***@***.***> | | Subject | Re: [apache/hadoop] HDFS-17358. EC: infinite lease recovery caused by the length of RWR equals to zero. (PR #6509) | @zhangshuyan0 commented on this pull request. Greate catch! I also reproduced this bug in my testing environment. If the client crashes when the amount of data written is less than 6 cells (use RS-6-3 for example), the file will never be able to be closed. If you could add a UT, that would be even better. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

haiyang1987 · 2024-01-29T11:33:40Z

https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockRecoveryWorker.java#L441

final long safeLength = getSafeLength(syncBlocks);
LOG.debug("Recovering block {}, length={}, safeLength={}, syncList={}", block,block.getNumBytes(), safeLength, syncBlocks);

if info.getNumBytes==0 ,then safeLength maybe as 0?
we can determine that if safeLength is 0 can remove this block？

hfutatzhanghb · 2024-01-29T11:45:53Z

https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockRecoveryWorker.java#L441
final long safeLength = getSafeLength(syncBlocks);
LOG.debug("Recovering block {}, length={}, safeLength={}, syncList={}", block,block.getNumBytes(), safeLength, syncBlocks);
if info.getNumBytes==0 ,then safeLength maybe as 0? we can determine that if safeLength is 0 can remove this block？

Hi, sir. Thanks for your reviewing. Yes, if info.getNumBytes==0 ,then safeLength is 0. I agree with you for using safeLength as the condition because it is more readable.

haiyang1987 · 2024-01-29T11:45:55Z

Contributor

Hi, sir. IIUC, it is safe. The reason is as below, original code is :
      ExtendedBlock newBlock = new ExtendedBlock(bpid, block.getBlockId(),
          safeLength, recoveryId);
      DatanodeProtocolClientSideTranslatorPB nn = getActiveNamenodeForBP(bpid);

      nn.commitBlockSynchronization(block, newBlock.getGenerationStamp(),
          newBlock.getNumBytes(), true, false, newLocs, newStorages);
We initialize an ExtendedBlock object named newBlock with safeLength equals to 0, then passing newBlock.getNumBytes() to method commitBlockSynchronization which means newlength.

yeah，I mean is it possible to determine if safeLength is 0, then directly call the logic of deleting this block?

if (safeLength == 0) {
        nn.commitBlockSynchronization(block, newBlock.getGenerationStamp(),
            newBlock.getNumBytes(), true, true, newLocs, newStorages);
        LOG.info("After block recovery, the length of new block is 0. " +
            "Will remove this block: {} from file.", newBlock);
        return;
 }

hadoop-yetus · 2024-01-29T14:07:56Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 23s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	32m 50s		trunk passed
+1 💚	compile	0m 39s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 38s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	checkstyle	0m 37s		trunk passed
+1 💚	mvnsite	0m 43s		trunk passed
+1 💚	javadoc	0m 41s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 3s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	1m 46s		trunk passed
+1 💚	shadedclient	20m 42s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 34s		the patch passed
+1 💚	compile	0m 37s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 37s		the patch passed
+1 💚	compile	0m 33s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	javac	0m 33s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 30s		the patch passed
+1 💚	mvnsite	0m 37s		the patch passed
+1 💚	javadoc	0m 29s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 58s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	1m 44s		the patch passed
+1 💚	shadedclient	20m 20s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	199m 56s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 28s		The patch does not generate ASF License warnings.
		287m 32s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6509/2/artifact/out/Dockerfile
GITHUB PR	#6509
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 33a97bbb8d75 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `8ef1574`
Default Java	Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6509/2/testReport/
Max. process+thread count	3981 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6509/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2024-01-29T16:16:16Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	17m 43s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	46m 28s		trunk passed
+1 💚	compile	1m 23s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	compile	1m 14s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	checkstyle	1m 11s		trunk passed
+1 💚	mvnsite	1m 22s		trunk passed
+1 💚	javadoc	1m 9s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 36s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	3m 18s		trunk passed
+1 💚	shadedclient	39m 43s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 13s		the patch passed
+1 💚	compile	1m 15s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javac	1m 15s		the patch passed
+1 💚	compile	1m 8s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	javac	1m 8s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 0s		the patch passed
+1 💚	mvnsite	1m 11s		the patch passed
+1 💚	javadoc	0m 54s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 25s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	3m 22s		the patch passed
+1 💚	shadedclient	39m 40s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	247m 48s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 41s		The patch does not generate ASF License warnings.
		415m 11s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestRollingUpgrade

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6509/3/artifact/out/Dockerfile
GITHUB PR	#6509
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 0574605d9246 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `8ef1574`
Default Java	Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6509/3/testReport/
Max. process+thread count	2846 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6509/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2024-01-29T16:28:13Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 20s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	30m 48s		trunk passed
+1 💚	compile	0m 42s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 36s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	checkstyle	0m 36s		trunk passed
+1 💚	mvnsite	0m 42s		trunk passed
+1 💚	javadoc	0m 39s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 4s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	1m 39s		trunk passed
+1 💚	shadedclient	20m 17s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 33s		the patch passed
+1 💚	compile	0m 35s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 35s		the patch passed
+1 💚	compile	0m 33s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	javac	0m 33s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 27s		the patch passed
+1 💚	mvnsite	0m 35s		the patch passed
+1 💚	javadoc	0m 28s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 1s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	1m 43s		the patch passed
+1 💚	shadedclient	20m 14s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	194m 45s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 27s		The patch does not generate ASF License warnings.
		279m 24s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6509/4/artifact/out/Dockerfile
GITHUB PR	#6509
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 290d3c2b0daf 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `5f436ea`
Default Java	Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6509/4/testReport/
Max. process+thread count	4196 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6509/4/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2024-01-30T08:44:41Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 20s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	31m 57s		trunk passed
+1 💚	compile	0m 41s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 35s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	checkstyle	0m 38s		trunk passed
+1 💚	mvnsite	0m 41s		trunk passed
+1 💚	javadoc	0m 41s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 1s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	1m 48s		trunk passed
+1 💚	shadedclient	20m 32s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 34s		the patch passed
+1 💚	compile	0m 38s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 38s		the patch passed
+1 💚	compile	0m 32s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	javac	0m 32s		the patch passed
-1 ❌	blanks	0m 0s	/blanks-eol.txt	The patch has 4 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
-0 ⚠️	checkstyle	0m 28s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 5 unchanged - 0 fixed = 6 total (was 5)
+1 💚	mvnsite	0m 36s		the patch passed
+1 💚	javadoc	0m 31s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 57s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	1m 40s		the patch passed
+1 💚	shadedclient	20m 17s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	200m 33s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 27s		The patch does not generate ASF License warnings.
		287m 9s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.datanode.TestDirectoryScanner
	hadoop.hdfs.TestLeaseRecoveryStriped
	hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6509/5/artifact/out/Dockerfile
GITHUB PR	#6509
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux fac5502bec56 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `96ecedc`
Default Java	Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6509/5/testReport/
Max. process+thread count	4687 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6509/5/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2024-01-30T08:51:29Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 20s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	31m 55s		trunk passed
+1 💚	compile	0m 40s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 36s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	checkstyle	0m 37s		trunk passed
+1 💚	mvnsite	0m 44s		trunk passed
+1 💚	javadoc	0m 41s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 3s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	1m 42s		trunk passed
+1 💚	shadedclient	20m 25s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 33s		the patch passed
+1 💚	compile	0m 36s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 36s		the patch passed
+1 💚	compile	0m 32s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	javac	0m 32s		the patch passed
-1 ❌	blanks	0m 0s	/blanks-eol.txt	The patch has 4 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
-0 ⚠️	checkstyle	0m 28s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 5 unchanged - 0 fixed = 6 total (was 5)
+1 💚	mvnsite	0m 37s		the patch passed
+1 💚	javadoc	0m 27s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 59s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	1m 39s		the patch passed
+1 💚	shadedclient	20m 3s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	205m 23s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 28s		The patch does not generate ASF License warnings.
		291m 41s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.datanode.TestDirectoryScanner
	hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy
	hadoop.hdfs.TestLeaseRecoveryStriped

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6509/8/artifact/out/Dockerfile
GITHUB PR	#6509
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux af6ba03f974e 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `96ecedc`
Default Java	Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6509/8/testReport/
Max. process+thread count	4807 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6509/8/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2024-01-30T09:58:25Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 32s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	41m 47s		trunk passed
+1 💚	compile	1m 19s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	compile	1m 15s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	checkstyle	1m 9s		trunk passed
+1 💚	mvnsite	1m 22s		trunk passed
+1 💚	javadoc	1m 5s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 38s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	3m 14s		trunk passed
+1 💚	shadedclient	34m 39s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 8s		the patch passed
+1 💚	compile	1m 12s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javac	1m 12s		the patch passed
+1 💚	compile	1m 4s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	javac	1m 4s		the patch passed
-1 ❌	blanks	0m 0s	/blanks-eol.txt	The patch has 4 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
-0 ⚠️	checkstyle	0m 56s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 5 unchanged - 0 fixed = 6 total (was 5)
+1 💚	mvnsite	1m 12s		the patch passed
+1 💚	javadoc	0m 53s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 33s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	3m 14s		the patch passed
+1 💚	shadedclient	34m 14s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	225m 0s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 42s		The patch does not generate ASF License warnings.
		359m 49s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestLeaseRecoveryStriped

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6509/6/artifact/out/Dockerfile
GITHUB PR	#6509
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux f19d056e8500 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `96ecedc`
Default Java	Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6509/6/testReport/
Max. process+thread count	3991 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6509/6/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2024-01-30T10:40:11Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 49s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	46m 24s		trunk passed
+1 💚	compile	1m 23s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	compile	1m 15s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	checkstyle	1m 11s		trunk passed
+1 💚	mvnsite	1m 23s		trunk passed
+1 💚	javadoc	1m 9s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 33s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	3m 19s		trunk passed
+1 💚	shadedclient	39m 29s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 10s		the patch passed
+1 💚	compile	1m 15s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javac	1m 15s		the patch passed
+1 💚	compile	1m 7s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	javac	1m 7s		the patch passed
-1 ❌	blanks	0m 0s	/blanks-eol.txt	The patch has 4 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
-0 ⚠️	checkstyle	1m 1s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 5 unchanged - 0 fixed = 6 total (was 5)
+1 💚	mvnsite	1m 13s		the patch passed
+1 💚	javadoc	0m 55s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 27s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	3m 22s		the patch passed
+1 💚	shadedclient	39m 46s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	249m 58s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 42s		The patch does not generate ASF License warnings.
		400m 47s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestLeaseRecoveryStriped
	hadoop.hdfs.TestRollingUpgrade

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6509/7/artifact/out/Dockerfile
GITHUB PR	#6509
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 26ad0358d483 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `96ecedc`
Default Java	Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6509/7/testReport/
Max. process+thread count	3048 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6509/7/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2024-01-30T10:47:23Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 49s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	47m 39s		trunk passed
+1 💚	compile	1m 23s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	compile	1m 13s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	checkstyle	1m 12s		trunk passed
+1 💚	mvnsite	1m 22s		trunk passed
+1 💚	javadoc	1m 8s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 35s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	3m 19s		trunk passed
+1 💚	shadedclient	39m 51s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 12s		the patch passed
+1 💚	compile	1m 16s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javac	1m 15s		the patch passed
+1 💚	compile	1m 6s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	javac	1m 6s		the patch passed
-1 ❌	blanks	0m 0s	/blanks-eol.txt	The patch has 4 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
-0 ⚠️	checkstyle	1m 1s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 5 unchanged - 0 fixed = 6 total (was 5)
+1 💚	mvnsite	1m 12s		the patch passed
+1 💚	javadoc	0m 55s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 27s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	3m 18s		the patch passed
+1 💚	shadedclient	39m 54s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	253m 49s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 43s		The patch does not generate ASF License warnings.
		405m 54s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestLeaseRecoveryStriped
	hadoop.hdfs.server.datanode.TestDirectoryScanner

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6509/9/artifact/out/Dockerfile
GITHUB PR	#6509
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 1f5d84414741 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `96ecedc`
Default Java	Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6509/9/testReport/
Max. process+thread count	3048 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6509/9/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hfutatzhanghb · 2024-02-20T08:49:33Z

@zhangshuyan0 @tasanuma @Hexiaoqiao Sir, code is ready for reviewing. Could you please help me review this PR when you have free time? Thanks a lot.

…quals to zero.

zhangshuyan0 · 2024-02-22T03:23:18Z

+        safeLength = getSafeLength(syncBlocks);
+      } else {
+        safeLength = 0;
+        LOG.warn("Block recovery: More than {} datanodes do not have the replica of block {}." +


Suggest printing out the value of zeroLenReplicaCnt as well.

Fixed it. Thanks sir.

What does this "More than" mean?

It seems useless here, have removed it . Originally, it was more than (9 - 6) datanodes ...

zhangshuyan0

+1. LGTM. Let's wait for yetus.

haiyang1987 · 2024-02-25T05:19:48Z

LGTM +1.

haiyang1987 · 2024-02-25T05:22:35Z

The result of the CI is here. https://ci-hadoop.apache.org/blue/organizations/jenkins/hadoop-multibranch/detail/PR-6509/26/tests
The failed tests seem not to be related.

hfutatzhanghb · 2024-02-26T02:43:04Z

@zhangshuyan0 @haiyang1987 @tasanuma @tomscut Sir, have updated unit test, please check it again~

tomscut · 2024-02-26T03:51:28Z

Good catch! The changes look good to me. Wait for the Jenkins.

hadoop-yetus · 2024-02-26T07:45:20Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 21s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	46m 52s		trunk passed
+1 💚	compile	0m 43s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 37s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	checkstyle	0m 40s		trunk passed
+1 💚	mvnsite	0m 45s		trunk passed
+1 💚	javadoc	0m 42s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 4s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	1m 45s		trunk passed
+1 💚	shadedclient	20m 45s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 34s		the patch passed
+1 💚	compile	0m 37s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 37s		the patch passed
+1 💚	compile	0m 33s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	javac	0m 33s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 27s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 5 unchanged - 0 fixed = 6 total (was 5)
+1 💚	mvnsite	0m 38s		the patch passed
+1 💚	javadoc	0m 31s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 57s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	1m 41s		the patch passed
+1 💚	shadedclient	20m 12s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	203m 14s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 32s		The patch does not generate ASF License warnings.
		305m 10s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.datanode.TestLargeBlockReport
	hadoop.hdfs.protocol.TestBlockListAsLongs
	hadoop.hdfs.tools.TestDFSAdmin

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6509/27/artifact/out/Dockerfile
GITHUB PR	#6509
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux f527b079541a 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `c7e3950`
Default Java	Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6509/27/testReport/
Max. process+thread count	5053 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6509/27/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2024-02-26T13:35:12Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 21s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	46m 12s		trunk passed
+1 💚	compile	0m 44s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 40s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	checkstyle	0m 38s		trunk passed
+1 💚	mvnsite	0m 46s		trunk passed
+1 💚	javadoc	0m 42s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 4s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	1m 40s		trunk passed
+1 💚	shadedclient	20m 28s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 36s		the patch passed
+1 💚	compile	0m 39s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 39s		the patch passed
+1 💚	compile	0m 33s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	javac	0m 33s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 30s		the patch passed
+1 💚	mvnsite	0m 35s		the patch passed
+1 💚	javadoc	0m 30s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 57s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	1m 42s		the patch passed
+1 💚	shadedclient	20m 32s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	197m 23s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 32s		The patch does not generate ASF License warnings.
		299m 5s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.datanode.TestLargeBlockReport
	hadoop.hdfs.server.datanode.TestDirectoryScanner
	hadoop.hdfs.protocol.TestBlockListAsLongs
	hadoop.hdfs.tools.TestDFSAdmin

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6509/28/artifact/out/Dockerfile
GITHUB PR	#6509
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 1895a134d100 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `5c79b13`
Default Java	Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6509/28/testReport/
Max. process+thread count	4003 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6509/28/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2024-02-26T17:40:46Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	17m 15s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	47m 37s		trunk passed
+1 💚	compile	1m 24s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	compile	1m 14s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	checkstyle	1m 12s		trunk passed
+1 💚	mvnsite	1m 22s		trunk passed
+1 💚	javadoc	1m 9s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 36s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	3m 21s		trunk passed
+1 💚	shadedclient	40m 28s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 12s		the patch passed
+1 💚	compile	1m 14s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javac	1m 14s		the patch passed
+1 💚	compile	1m 6s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	javac	1m 6s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 1s		the patch passed
+1 💚	mvnsite	1m 12s		the patch passed
+1 💚	javadoc	0m 55s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 30s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	3m 21s		the patch passed
+1 💚	shadedclient	40m 8s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	253m 32s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 43s		The patch does not generate ASF License warnings.
		423m 12s

Reason	Tests
Failed junit tests	hadoop.hdfs.protocol.TestBlockListAsLongs
	hadoop.hdfs.tools.TestDFSAdmin
	hadoop.hdfs.server.datanode.TestLargeBlockReport

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6509/29/artifact/out/Dockerfile
GITHUB PR	#6509
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux c38cac31b717 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `b0b1ef7`
Default Java	Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6509/29/testReport/
Max. process+thread count	3048 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6509/29/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hfutatzhanghb · 2024-02-27T02:03:54Z

The failed UTs were all passed in my local.

zhangshuyan0 · 2024-02-27T02:21:21Z

Committed to trunk. Thanks for your contributions! @hfutatzhanghb @haiyang1987 @tomscut

tomscut · 2024-02-29T11:12:14Z

@zhangshuyan0 Hi, I closed the issue HDFS-17358.

…quals to zero or datanode does not have the replica. (apache#6509). Contributed by farmmamba. Reviewed-by: Tao Li <tomscut@apache.org> Reviewed-by: Haiyang Hu <haiyang.hu@shopee.com> Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>

github-actions Bot added HDFS trunk labels Jan 28, 2024

hfutatzhanghb force-pushed the HDFS-17358 branch 2 times, most recently from 475304b to 8ef1574 Compare January 29, 2024 09:19

zhangshuyan0 reviewed Jan 29, 2024

View reviewed changes

hfutatzhanghb force-pushed the HDFS-17358 branch from 8ef1574 to 5f436ea Compare January 29, 2024 11:47

hfutatzhanghb force-pushed the HDFS-17358 branch 6 times, most recently from 96ecedc to 6c1081a Compare January 30, 2024 08:03

hfutatzhanghb marked this pull request as draft January 30, 2024 08:43

hfutatzhanghb force-pushed the HDFS-17358 branch from 6c1081a to 64d91bc Compare January 30, 2024 09:18

hfutatzhanghb force-pushed the HDFS-17358 branch 2 times, most recently from 4619de2 to 37b289f Compare January 30, 2024 13:10

hfutatzhanghb force-pushed the HDFS-17358 branch from 36eae19 to 3a54b3d Compare February 20, 2024 08:32

HDFS-17358. EC: infinite lease recovery caused by the length of RWR e…

3eeacd6

…quals to zero.

hfutatzhanghb force-pushed the HDFS-17358 branch from 3a54b3d to 3eeacd6 Compare February 21, 2024 07:40

hfutatzhanghb closed this Feb 21, 2024

hfutatzhanghb reopened this Feb 21, 2024

zhangshuyan0 reviewed Feb 22, 2024

View reviewed changes

hfutatzhanghb added 2 commits February 22, 2024 11:45

Print dnNotHaveReplicaCnt in logs.

f3fae9b

Modify logs.

f78b3df

zhangshuyan0 approved these changes Feb 22, 2024

View reviewed changes

haiyang1987 approved these changes Feb 25, 2024

View reviewed changes

modify unit tests.

c7e3950

hfutatzhanghb changed the title ~~HDFS-17358. EC: infinite lease recovery caused by the length of RWR equals to zero.~~ HDFS-17358. EC: infinite lease recovery caused by the length of RWR equals to zero or datanode does not have replica. Feb 26, 2024

fix checkstyle warning.

b0b1ef7

hfutatzhanghb force-pushed the HDFS-17358 branch from 5c79b13 to b0b1ef7 Compare February 26, 2024 10:36

tomscut approved these changes Feb 27, 2024

View reviewed changes

hfutatzhanghb changed the title ~~HDFS-17358. EC: infinite lease recovery caused by the length of RWR equals to zero or datanode does not have replica.~~ HDFS-17358. EC: infinite lease recovery caused by the length of RWR equals to zero or datanode does not have the replica. Feb 27, 2024

zhangshuyan0 merged commit 15af529 into apache:trunk Feb 27, 2024

hfutatzhanghb deleted the HDFS-17358 branch February 27, 2024 02:21

Conversation

hfutatzhanghb commented Jan 28, 2024

Description of PR

Uh oh!

hadoop-yetus commented Jan 28, 2024

Uh oh!

hfutatzhanghb commented Jan 29, 2024

Uh oh!

zhangshuyan0 left a comment

Choose a reason for hiding this comment

Uh oh!

hfutatzhanghb commented Jan 29, 2024 via email

Uh oh!

haiyang1987 commented Jan 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hfutatzhanghb commented Jan 29, 2024

Uh oh!

haiyang1987 commented Jan 29, 2024

Uh oh!

hadoop-yetus commented Jan 29, 2024

Uh oh!

hadoop-yetus commented Jan 29, 2024

Uh oh!

hadoop-yetus commented Jan 29, 2024

Uh oh!

hadoop-yetus commented Jan 30, 2024

Uh oh!

hadoop-yetus commented Jan 30, 2024

Uh oh!

hadoop-yetus commented Jan 30, 2024

Uh oh!

hadoop-yetus commented Jan 30, 2024

Uh oh!

hadoop-yetus commented Jan 30, 2024

Uh oh!

hfutatzhanghb commented Feb 20, 2024

Uh oh!

zhangshuyan0 Feb 22, 2024

Choose a reason for hiding this comment

Uh oh!

hfutatzhanghb Feb 22, 2024

Choose a reason for hiding this comment

Uh oh!

zhangshuyan0 Feb 22, 2024

Choose a reason for hiding this comment

Uh oh!

hfutatzhanghb Feb 22, 2024

Choose a reason for hiding this comment

Uh oh!

zhangshuyan0 left a comment

Choose a reason for hiding this comment

Uh oh!

haiyang1987 commented Feb 25, 2024

Uh oh!

haiyang1987 commented Feb 25, 2024

Uh oh!

hfutatzhanghb commented Feb 26, 2024

Uh oh!

tomscut commented Feb 26, 2024

Uh oh!

hadoop-yetus commented Feb 26, 2024

Uh oh!

hadoop-yetus commented Feb 26, 2024

Uh oh!

hadoop-yetus commented Feb 26, 2024

Uh oh!

hfutatzhanghb commented Feb 27, 2024

Uh oh!

zhangshuyan0 commented Feb 27, 2024

Uh oh!

tomscut commented Feb 29, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

haiyang1987 commented Jan 29, 2024 •

edited

Loading