HADOOP-19187: [ABFS][FNSOverBlob][Backport to 3.4] AbfsClient Refactoring to Support Multiple Implementation of Clients. #7198

anujmodi2021 · 2024-12-02T09:15:04Z

This is the first PR in the series of work done under Parent Jira: HADOOP-19179
Jira for this Patch: https://issues.apache.org/jira/browse/HADOOP-19187

Original PR was merged to trunk and had enough bake time: #6879

Cherry-picked commit: b15ed27

Original Description

How was this patch tested?

Test suite was ran.
Results added below

…ultiple Implementation of Clients. (apache#6879) Refactor AbfsClient into DFS and Blob Client. Contributed by Anuj Modi

anujmodi2021 · 2024-12-02T09:15:50Z

:::: AGGREGATED TEST RESULT ::::

============================================================
HNS-OAuth

[WARNING] Tests run: 154, Failures: 0, Errors: 0, Skipped: 2
[WARNING] Tests run: 649, Failures: 0, Errors: 0, Skipped: 82
[WARNING] Tests run: 171, Failures: 0, Errors: 0, Skipped: 25
[WARNING] Tests run: 262, Failures: 0, Errors: 0, Skipped: 23

============================================================
HNS-SharedKey

[WARNING] Tests run: 154, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 649, Failures: 0, Errors: 0, Skipped: 34
[WARNING] Tests run: 171, Failures: 0, Errors: 0, Skipped: 25
[WARNING] Tests run: 262, Failures: 0, Errors: 0, Skipped: 10

============================================================
NonHNS-SharedKey

[WARNING] Tests run: 154, Failures: 0, Errors: 0, Skipped: 9
[WARNING] Tests run: 633, Failures: 0, Errors: 0, Skipped: 274
[WARNING] Tests run: 171, Failures: 0, Errors: 0, Skipped: 27
[WARNING] Tests run: 262, Failures: 0, Errors: 0, Skipped: 11

============================================================
AppendBlob-HNS-OAuth

[WARNING] Tests run: 154, Failures: 0, Errors: 0, Skipped: 2
[WARNING] Tests run: 649, Failures: 0, Errors: 0, Skipped: 84
[WARNING] Tests run: 171, Failures: 0, Errors: 0, Skipped: 49
[WARNING] Tests run: 262, Failures: 0, Errors: 0, Skipped: 23

Time taken: 111 mins 59 secs.

anujmodi2021 · 2024-12-02T09:17:59Z

@mukund-thakur
This is the back port PR for 3.4.
This change had enough bake time in trunk. Have done the checks and test suite also run fine.

This should be good to merge.
Thanks

hadoop-yetus · 2024-12-02T11:34:00Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	16m 57s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+0 🆗	markdownlint	0m 1s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 5 new or modified test files.
			_ branch-3.4 Compile Tests _
+1 💚	mvninstall	42m 25s		branch-3.4 passed
+1 💚	compile	0m 39s		branch-3.4 passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	compile	0m 36s		branch-3.4 passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	checkstyle	0m 34s		branch-3.4 passed
+1 💚	mvnsite	0m 42s		branch-3.4 passed
+1 💚	javadoc	0m 38s		branch-3.4 passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 34s		branch-3.4 passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	1m 6s		branch-3.4 passed
+1 💚	shadedclient	32m 36s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 29s		the patch passed
+1 💚	compile	0m 29s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javac	0m 29s		the patch passed
+1 💚	compile	0m 27s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	javac	0m 27s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 19s	/results-checkstyle-hadoop-tools_hadoop-azure.txt	hadoop-tools/hadoop-azure: The patch generated 7 new + 15 unchanged - 3 fixed = 22 total (was 18)
+1 💚	mvnsite	0m 30s		the patch passed
+1 💚	javadoc	0m 26s		hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 generated 0 new + 11 unchanged - 4 fixed = 11 total (was 15)
+1 💚	javadoc	0m 25s		hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga generated 0 new + 11 unchanged - 4 fixed = 11 total (was 15)
+1 💚	spotbugs	1m 2s		the patch passed
+1 💚	shadedclient	32m 16s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 17s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 37s		The patch does not generate ASF License warnings.
		137m 37s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7198/1/artifact/out/Dockerfile
GITHUB PR	#7198
Optional Tests	dupname asflicense codespell detsecrets xmllint compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle markdownlint
uname	Linux cf9c74f12ea0 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.4 / `a7f07e7`
Default Java	Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7198/1/testReport/
Max. process+thread count	620 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7198/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

mukund-thakur · 2024-12-02T21:15:24Z

@saikatroy038 can you please take a look at this?

mukund-thakur · 2024-12-03T19:57:24Z

Also fix the checkstyle @anujmodi2021

saikatroy038

LGTM

hadoop-yetus · 2024-12-17T08:05:34Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	16m 37s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 5 new or modified test files.
			_ branch-3.4 Compile Tests _
+1 💚	mvninstall	36m 38s		branch-3.4 passed
+1 💚	compile	0m 38s		branch-3.4 passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	compile	0m 35s		branch-3.4 passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	checkstyle	0m 35s		branch-3.4 passed
+1 💚	mvnsite	0m 41s		branch-3.4 passed
+1 💚	javadoc	0m 42s		branch-3.4 passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 35s		branch-3.4 passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	1m 8s		branch-3.4 passed
+1 💚	shadedclient	32m 41s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 28s		the patch passed
+1 💚	compile	0m 30s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javac	0m 30s		the patch passed
+1 💚	compile	0m 27s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	javac	0m 27s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 21s	/results-checkstyle-hadoop-tools_hadoop-azure.txt	hadoop-tools/hadoop-azure: The patch generated 1 new + 15 unchanged - 3 fixed = 16 total (was 18)
+1 💚	mvnsite	0m 30s		the patch passed
+1 💚	javadoc	0m 26s		hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 generated 0 new + 11 unchanged - 4 fixed = 11 total (was 15)
+1 💚	javadoc	0m 24s		hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga generated 0 new + 11 unchanged - 4 fixed = 11 total (was 15)
+1 💚	spotbugs	1m 3s		the patch passed
+1 💚	shadedclient	32m 15s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 18s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 38s		The patch does not generate ASF License warnings.
		131m 33s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7198/2/artifact/out/Dockerfile
GITHUB PR	#7198
Optional Tests	dupname asflicense codespell detsecrets xmllint compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle markdownlint
uname	Linux 12e006b2465c 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.4 / `0558e10`
Default Java	Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7198/2/testReport/
Max. process+thread count	648 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7198/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

anujmodi2021 · 2024-12-23T06:40:41Z

@mukund-thakur
Fixed the checkstye except a Todo which is also there on trunk. It will be removed in subsequent PR under FNS-Blob.

anujmodi2021 and others added 2 commits November 28, 2024 20:44

HADOOP-19187: [ABFS][FNSOverBlob] AbfsClient Refactoring to Support M…

3707dfe

…ultiple Implementation of Clients. (apache#6879) Refactor AbfsClient into DFS and Blob Client. Contributed by Anuj Modi

Fixing Test Failures

a7f07e7

anujmodi2021 changed the title ~~Branch 3.4 hadoop 19187~~ HADOOP-19187: [ABFS][FNSOverBlob][Backport to 3.4] AbfsClient Refactoring to Support Multiple Implementation of Clients. Dec 2, 2024

saikatroy038 approved these changes Dec 5, 2024

View reviewed changes

Unused Imports

0558e10

manika137 mentioned this pull request Dec 19, 2024

HADOOP-19357. [ABFS] [Backport to 3.4]: Optimizations for Retry Handling and Throttling #7234

Merged

anujmodi2021 merged commit fcefb62 into apache:branch-3.4 Dec 26, 2024
3 checks passed

anujmodi2021 deleted the branch-3.4_HADOOP-19187 branch December 26, 2024 05:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HADOOP-19187: [ABFS][FNSOverBlob][Backport to 3.4] AbfsClient Refactoring to Support Multiple Implementation of Clients. #7198

HADOOP-19187: [ABFS][FNSOverBlob][Backport to 3.4] AbfsClient Refactoring to Support Multiple Implementation of Clients. #7198

Uh oh!

anujmodi2021 commented Dec 2, 2024

Uh oh!

anujmodi2021 commented Dec 2, 2024

Uh oh!

anujmodi2021 commented Dec 2, 2024 •

edited

Loading

Uh oh!

hadoop-yetus commented Dec 2, 2024

Uh oh!

mukund-thakur commented Dec 2, 2024

Uh oh!

mukund-thakur commented Dec 3, 2024

Uh oh!

saikatroy038 left a comment

Uh oh!

hadoop-yetus commented Dec 17, 2024

Uh oh!

anujmodi2021 commented Dec 23, 2024

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

HADOOP-19187: [ABFS][FNSOverBlob][Backport to 3.4] AbfsClient Refactoring to Support Multiple Implementation of Clients. #7198

HADOOP-19187: [ABFS][FNSOverBlob][Backport to 3.4] AbfsClient Refactoring to Support Multiple Implementation of Clients. #7198

Uh oh!

Conversation

anujmodi2021 commented Dec 2, 2024

Original PR was merged to trunk and had enough bake time: #6879

How was this patch tested?

Uh oh!

anujmodi2021 commented Dec 2, 2024

============================================================ HNS-OAuth

============================================================ HNS-SharedKey

============================================================ NonHNS-SharedKey

============================================================ AppendBlob-HNS-OAuth

Uh oh!

anujmodi2021 commented Dec 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hadoop-yetus commented Dec 2, 2024

Uh oh!

mukund-thakur commented Dec 2, 2024

Uh oh!

mukund-thakur commented Dec 3, 2024

Uh oh!

saikatroy038 left a comment

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented Dec 17, 2024

Uh oh!

anujmodi2021 commented Dec 23, 2024

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

============================================================
HNS-OAuth

============================================================
HNS-SharedKey

============================================================
NonHNS-SharedKey

============================================================
AppendBlob-HNS-OAuth

anujmodi2021 commented Dec 2, 2024 •

edited

Loading