HADOOP-19357. [ABFS] [Backport to 3.4]: Optimizations for Retry Handling and Throttling

[manika137]

Description of PR

Commit on trunk: fc42da7
PR on trunk: #7216
Target Jira: https://issues.apache.org/jira/browse/HADOOP-19357

Default for following configs are changed:

Client-side throttling (CST): Off
Client Backoff - 500ms (reduced from 3sec)
Max Backoff - 25s (reduced from 30sec)
Min Backoff - 500ms (reduced from 3sec)

Contributed by Manika Joshi (@manika137)

How was this patch tested

Test suite run result are pasted in the comment below.
The failing test is known and is fixed in #7198

[manika137]

Aggregated Test Results

============================================================
HNS-OAuth

[WARNING] Tests run: 154, Failures: 0, Errors: 0, Skipped: 2
[WARNING] Tests run: 648, Failures: 0, Errors: 0, Skipped: 82
[WARNING] Tests run: 171, Failures: 0, Errors: 0, Skipped: 25
[WARNING] Tests run: 262, Failures: 0, Errors: 0, Skipped: 32

============================================================
HNS-SharedKey

[WARNING] Tests run: 154, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 648, Failures: 0, Errors: 0, Skipped: 34
[WARNING] Tests run: 171, Failures: 0, Errors: 0, Skipped: 25
[WARNING] Tests run: 262, Failures: 0, Errors: 0, Skipped: 19

============================================================
NonHNS-SharedKey

[ERROR] testCheckAccessForAccountWithoutNS(org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCheckAccess) Time elapsed: 1.327 s <<< FAILURE!

[WARNING] Tests run: 154, Failures: 0, Errors: 0, Skipped: 9
[ERROR] Tests run: 627, Failures: 1, Errors: 0, Skipped: 272
[WARNING] Tests run: 171, Failures: 0, Errors: 0, Skipped: 27
[WARNING] Tests run: 262, Failures: 0, Errors: 0, Skipped: 20

============================================================
AppendBlob-HNS-OAuth

[WARNING] Tests run: 154, Failures: 0, Errors: 0, Skipped: 2
[WARNING] Tests run: 648, Failures: 0, Errors: 0, Skipped: 84
[WARNING] Tests run: 171, Failures: 0, Errors: 0, Skipped: 49
[WARNING] Tests run: 262, Failures: 0, Errors: 0, Skipped: 32

[hadoop-yetus]

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	6m 28s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ branch-3.4 Compile Tests _
+1 💚	mvninstall	23m 6s		branch-3.4 passed
+1 💚	compile	0m 24s		branch-3.4 passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	compile	0m 21s		branch-3.4 passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚	checkstyle	0m 21s		branch-3.4 passed
+1 💚	mvnsite	0m 25s		branch-3.4 passed
+1 💚	javadoc	0m 27s		branch-3.4 passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 21s		branch-3.4 passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚	spotbugs	0m 42s		branch-3.4 passed
+1 💚	shadedclient	19m 11s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 16s		the patch passed
+1 💚	compile	0m 17s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javac	0m 17s		the patch passed
+1 💚	compile	0m 15s		the patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚	javac	0m 15s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 13s		the patch passed
+1 💚	mvnsite	0m 17s		the patch passed
+1 💚	javadoc	0m 16s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 17s		the patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚	spotbugs	0m 43s		the patch passed
+1 💚	shadedclient	19m 10s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	1m 57s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 24s		The patch does not generate ASF License warnings.
		76m 33s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7234/1/artifact/out/Dockerfile
GITHUB PR	#7234
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 2d174dfd9ba9 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.4 / bcdf978
Default Java	Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7234/1/testReport/
Max. process+thread count	674 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7234/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.