HADOOP-19472: [ABFS] Remove write aggressiveness optimization #8141

anmolanmol1234 · 2025-12-19T10:37:17Z

Removing write aggressiveness code and related metrics from trunk.
Performance analysis showed that write throughput was already saturated as a result of previously merged write-path optimizations. Consequently, this change does not provide measurable gains for write operations.

The PR that introduced these changes was:- #7669

anujmodi2021 · 2025-12-23T09:04:14Z

...op-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/FileSystemConfigurations.java

I know this is not a part of change here but since we are doing code cleanup here, I feel we should keep these constants in some constant file only like InternalConstants or MetricsConstants.
This file mainly contains constants used in configuration.

Starting from ZERO to SCALE_DIRECTION_NO_ACTION_NEEDED

anujmodi2021 · 2025-12-23T09:16:53Z

...op-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/FileSystemConfigurations.java

   * Number of bytes in a gigabyte.
   */
  public static final long BYTES_PER_GIGABYTE = 1024L * 1024 * 1024;
-  /**


There are a few constants in AbfsHttpConstants file that also need to be removed.

anujmodi2021 · 2025-12-23T09:22:04Z

...c/main/java/org/apache/hadoop/fs/azurebfs/enums/AbfsWriteResourceUtilizationMetricsEnum.java

@@ -1,107 +0,0 @@
-/**


Sine there is only one resource utilization metric left. Do we need the base class?
Can we simply have one stand alone class called ResourceUtilizationMetrics instead of saying read/write?

Also anyway we are monitoring JVM resources here, so single metric is sufficient IMO.

I created this design to be generic such that in future if we want to add metrics for any other operation that can be done in a separate class as the update method takes stats specific to a particular operation.

anmolanmol1234 · 2025-12-30T06:05:00Z

============================================================
HNS-OAuth-DFS

[WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 892, Failures: 0, Errors: 0, Skipped: 223
[WARNING] Tests run: 158, Failures: 0, Errors: 0, Skipped: 8
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 23

============================================================
HNS-SharedKey-DFS

[WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 5
[WARNING] Tests run: 895, Failures: 0, Errors: 0, Skipped: 169
[WARNING] Tests run: 158, Failures: 0, Errors: 0, Skipped: 8
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 10

============================================================
NonHNS-SharedKey-DFS

[WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 11
[WARNING] Tests run: 732, Failures: 0, Errors: 0, Skipped: 290
[WARNING] Tests run: 158, Failures: 0, Errors: 0, Skipped: 9
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 11

============================================================
AppendBlob-HNS-OAuth-DFS

[WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 892, Failures: 0, Errors: 0, Skipped: 234
[WARNING] Tests run: 135, Failures: 0, Errors: 0, Skipped: 9
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 23

============================================================
NonHNS-SharedKey-Blob

[WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 11
[WARNING] Tests run: 740, Failures: 0, Errors: 0, Skipped: 147
[WARNING] Tests run: 158, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 11

============================================================
NonHNS-OAuth-DFS

[WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 11
[WARNING] Tests run: 729, Failures: 0, Errors: 0, Skipped: 292
[WARNING] Tests run: 158, Failures: 0, Errors: 0, Skipped: 9
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 24

============================================================
NonHNS-OAuth-Blob

[WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 11
[WARNING] Tests run: 737, Failures: 0, Errors: 0, Skipped: 159
[WARNING] Tests run: 158, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 24

============================================================
AppendBlob-NonHNS-OAuth-Blob

[WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 11
[WARNING] Tests run: 732, Failures: 0, Errors: 0, Skipped: 205
[WARNING] Tests run: 135, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 24

============================================================
HNS-Oauth-DFS-IngressBlob

[WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 766, Failures: 0, Errors: 0, Skipped: 232
[WARNING] Tests run: 158, Failures: 0, Errors: 0, Skipped: 8
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 23

============================================================
NonHNS-OAuth-DFS-IngressBlob

[WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 11
[WARNING] Tests run: 729, Failures: 0, Errors: 0, Skipped: 289
[WARNING] Tests run: 158, Failures: 0, Errors: 0, Skipped: 9
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 24

… HADOOP-19472_fix

hadoop-yetus · 2025-12-30T08:14:32Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 16s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
-1 ❌	mvninstall	42m 15s	/branch-mvninstall-root.txt	root in trunk failed.
+1 💚	compile	0m 44s		trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 46s		trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚	checkstyle	0m 34s		trunk passed
+1 💚	mvnsite	0m 51s		trunk passed
+1 💚	javadoc	0m 41s		trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 37s		trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
-1 ❌	spotbugs	1m 23s	/branch-spotbugs-hadoop-tools_hadoop-azure-warnings.html	hadoop-tools/hadoop-azure in trunk has 1 extant spotbugs warnings.
+1 💚	shadedclient	31m 13s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 38s		the patch passed
+1 💚	compile	0m 36s		the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 36s		the patch passed
+1 💚	compile	0m 38s		the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 38s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 22s		the patch passed
+1 💚	mvnsite	0m 42s		the patch passed
-1 ❌	javadoc	0m 30s	/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt	hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04 with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 generated 5 new + 1555 unchanged - 30 fixed = 1560 total (was 1585)
-1 ❌	javadoc	0m 30s	/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt	hadoop-tools_hadoop-azure-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04 with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 generated 4 new + 1435 unchanged - 29 fixed = 1439 total (was 1464)
+1 💚	spotbugs	1m 23s		the patch passed
+1 💚	shadedclient	31m 44s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	3m 19s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 34s		The patch does not generate ASF License warnings.
		122m 54s

Subsystem	Report/Notes
Docker	ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8141/3/artifact/out/Dockerfile
GITHUB PR	#8141
JIRA Issue	HADOOP-19472
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux d821473746d1 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `ff48ed3`
Default Java	Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Multi-JDK versions	/usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8141/3/testReport/
Max. process+thread count	649 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8141/3/console
versions	git=2.25.1 maven=3.9.11 spotbugs=4.9.7
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

anujmodi2021

Thanks for taking my comments.
+1 LGTM

bhattmanish98

+1

hadoop-yetus · 2026-01-14T09:47:57Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	22m 6s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	43m 16s		trunk passed
+1 💚	compile	0m 44s		trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 45s		trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚	checkstyle	0m 33s		trunk passed
+1 💚	mvnsite	0m 50s		trunk passed
-1 ❌	javadoc	0m 42s	/branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt	hadoop-azure in trunk failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04.
-1 ❌	javadoc	0m 38s	/branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt	hadoop-azure in trunk failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04.
-1 ❌	spotbugs	1m 25s	/branch-spotbugs-hadoop-tools_hadoop-azure-warnings.html	hadoop-tools/hadoop-azure in trunk has 1 extant spotbugs warnings.
+1 💚	shadedclient	31m 22s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	31m 45s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 41s		the patch passed
+1 💚	compile	0m 35s		the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 35s		the patch passed
+1 💚	compile	0m 39s		the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 39s		the patch passed
+1 💚	blanks	0m 1s		The patch has no blanks issues.
+1 💚	checkstyle	0m 22s		the patch passed
+1 💚	mvnsite	0m 41s		the patch passed
-1 ❌	javadoc	0m 30s	/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt	hadoop-azure in the patch failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04.
-1 ❌	javadoc	0m 30s	/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt	hadoop-azure in the patch failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04.
+1 💚	spotbugs	1m 25s		the patch passed
+1 💚	shadedclient	32m 5s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	3m 23s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 35s		The patch does not generate ASF License warnings.
		145m 45s

Subsystem	Report/Notes
Docker	ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8141/4/artifact/out/Dockerfile
GITHUB PR	#8141
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 8b12118ddc40 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `9ecff93`
Default Java	Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Multi-JDK versions	/usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8141/4/testReport/
Max. process+thread count	570 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8141/4/console
versions	git=2.25.1 maven=3.9.11 spotbugs=4.9.7
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Remove write aggressiveness code

6ba355b

This comment was marked as outdated.

Sign in to view

github-actions bot added trunk TOOLS ABFS labels Dec 19, 2025

This comment was marked as outdated.

Sign in to view

anujmodi2021 reviewed Dec 23, 2025

View reviewed changes

PR comments

cd7da4c

This comment was marked as outdated.

Sign in to view

manika137 approved these changes Dec 30, 2025

View reviewed changes

Merge branch 'trunk' of https://github.com/anmolanmol1234/hadoop into…

ff48ed3

… HADOOP-19472_fix

anujmodi2021 approved these changes Jan 5, 2026

View reviewed changes

bhattmanish98 approved these changes Jan 14, 2026

View reviewed changes

Merge conflicts

9ecff93

HADOOP-19472: [ABFS] Remove write aggressiveness optimization #8141

Are you sure you want to change the base?

HADOOP-19472: [ABFS] Remove write aggressiveness optimization #8141

Uh oh!

Conversation

anmolanmol1234 commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

anujmodi2021 Dec 23, 2025

Choose a reason for hiding this comment

Uh oh!

anmolanmol1234 Dec 29, 2025

Choose a reason for hiding this comment

Uh oh!

anujmodi2021 Dec 23, 2025

Choose a reason for hiding this comment

Uh oh!

anmolanmol1234 Dec 29, 2025

Choose a reason for hiding this comment

Uh oh!

anujmodi2021 Dec 23, 2025

Choose a reason for hiding this comment

Uh oh!

anmolanmol1234 Dec 29, 2025

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

anmolanmol1234 commented Dec 30, 2025

============================================================ HNS-OAuth-DFS

============================================================ HNS-SharedKey-DFS

============================================================ NonHNS-SharedKey-DFS

============================================================ AppendBlob-HNS-OAuth-DFS

============================================================ NonHNS-SharedKey-Blob

============================================================ NonHNS-OAuth-DFS

============================================================ NonHNS-OAuth-Blob

============================================================ AppendBlob-NonHNS-OAuth-Blob

============================================================ HNS-Oauth-DFS-IngressBlob

============================================================ NonHNS-OAuth-DFS-IngressBlob

Uh oh!

hadoop-yetus commented Dec 30, 2025

Uh oh!

anujmodi2021 left a comment

Choose a reason for hiding this comment

Uh oh!

bhattmanish98 left a comment

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented Jan 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

anmolanmol1234 commented Dec 19, 2025 •

edited

Loading

============================================================
HNS-OAuth-DFS

============================================================
HNS-SharedKey-DFS

============================================================
NonHNS-SharedKey-DFS

============================================================
AppendBlob-HNS-OAuth-DFS

============================================================
NonHNS-SharedKey-Blob

============================================================
NonHNS-OAuth-DFS

============================================================
NonHNS-OAuth-Blob

============================================================
AppendBlob-NonHNS-OAuth-Blob

============================================================
HNS-Oauth-DFS-IngressBlob

============================================================
NonHNS-OAuth-DFS-IngressBlob