HADOOP-18797. Support Concurrent Writes With S3A Magic Committer

[shameersss1]

Jobs which commit their work to S3 through
magic committer now use a unique magic
containing the job ID:
__magic_job-${jobid}

This allows for multiple jobs to write
to the same destination simultaneously.

Contributed by Syed Shameerur Rahman

Description of PR

How was this patch tested?

For code changes:

• Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
• Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

[shameersss1]

FYI: @steveloughran

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	7m 9s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 11 new or modified test files.
			_ branch-3.3 Compile Tests _
-1 ❌	mvninstall	46m 43s	/branch-mvninstall-root.txt	root in branch-3.3 failed.
+1 💚	compile	0m 42s		branch-3.3 passed
+1 💚	checkstyle	0m 36s		branch-3.3 passed
+1 💚	mvnsite	0m 51s		branch-3.3 passed
+1 💚	javadoc	0m 41s		branch-3.3 passed
+1 💚	spotbugs	1m 17s		branch-3.3 passed
-1 ❌	shadedclient	37m 13s		branch has errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 41s		the patch passed
+1 💚	compile	0m 33s		the patch passed
+1 💚	javac	0m 33s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 22s	/results-checkstyle-hadoop-tools_hadoop-aws.txt	hadoop-tools/hadoop-aws: The patch generated 1 new + 10 unchanged - 0 fixed = 11 total (was 10)
+1 💚	mvnsite	0m 37s		the patch passed
+1 💚	javadoc	0m 27s		the patch passed
+1 💚	spotbugs	1m 11s		the patch passed
-1 ❌	shadedclient	37m 6s		patch has errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 46s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 40s		The patch does not generate ASF License warnings.
		143m 18s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6122/1/artifact/out/Dockerfile
GITHUB PR	#6122
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
uname	Linux d6b81cc3fc00 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / 8f13118
Default Java	Private Build-1.8.0_362-8u372-gaus1-0ubuntu118.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6122/1/testReport/
Max. process+thread count	556 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6122/1/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	11m 47s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 11 new or modified test files.
			_ branch-3.3 Compile Tests _
+1 💚	mvninstall	49m 48s		branch-3.3 passed
+1 💚	compile	0m 39s		branch-3.3 passed
+1 💚	checkstyle	0m 32s		branch-3.3 passed
+1 💚	mvnsite	0m 51s		branch-3.3 passed
+1 💚	javadoc	0m 38s		branch-3.3 passed
+1 💚	spotbugs	1m 14s		branch-3.3 passed
+1 💚	shadedclient	37m 37s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 40s		the patch passed
+1 💚	compile	0m 32s		the patch passed
+1 💚	javac	0m 32s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 22s	/results-checkstyle-hadoop-tools_hadoop-aws.txt	hadoop-tools/hadoop-aws: The patch generated 1 new + 10 unchanged - 0 fixed = 11 total (was 10)
+1 💚	mvnsite	0m 38s		the patch passed
+1 💚	javadoc	0m 27s		the patch passed
+1 💚	spotbugs	1m 9s		the patch passed
+1 💚	shadedclient	36m 31s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 43s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 39s		The patch does not generate ASF License warnings.
		150m 55s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6122/1/artifact/out/Dockerfile
GITHUB PR	#6122
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
uname	Linux 32b24fe9347a 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / 8f13118
Default Java	Private Build-1.8.0_362-8u372-gaus1-0ubuntu118.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6122/1/testReport/
Max. process+thread count	726 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6122/1/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[steveloughran]

looks good and I'm not going to review backports except for backporting issues.

which s3 region did you test against, and with what parameters?

[shameersss1]

looks good and I'm not going to review backports except for backporting issues.

which s3 region did you test against, and with what parameters?

How was this patch tested?

1. Ran S3A Unit Tests

2. Ran S3A Integration test in us-west-1 region

[INFO] Running org.apache.hadoop.fs.s3a.commit.staging.TestDirectoryCommitterScale [INFO] Running org.apache.hadoop.fs.s3a.commit.staging.TestPaths [INFO] Running org.apache.hadoop.fs.s3a.commit.TestMagicCommitPaths [INFO] Running org.apache.hadoop.fs.s3a.commit.staging.TestStagingCommitter [INFO] Running org.apache.hadoop.fs.s3a.commit.staging.TestStagingPartitionedFileListing [INFO] Running org.apache.hadoop.fs.s3a.commit.staging.TestStagingDirectoryOutputCommitter [INFO] Running org.apache.hadoop.fs.s3a.commit.staging.TestStagingPartitionedJobCommit [INFO] Running org.apache.hadoop.fs.s3a.commit.staging.TestStagingPartitionedTaskCommit [INFO] Tests run: 28, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.473 s - in org.apache.hadoop.fs.s3a.commit.TestMagicCommitPaths [INFO] Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.611 s - in org.apache.hadoop.fs.s3a.commit.staging.TestPaths [INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.965 s - in org.apache.hadoop.fs.s3a.commit.staging.TestStagingDirectoryOutputCommitter [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.474 s - in org.apache.hadoop.fs.s3a.commit.staging.TestStagingPartitionedFileListing [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.627 s - in org.apache.hadoop.fs.s3a.commit.staging.TestStagingPartitionedJobCommit [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.249 s - in org.apache.hadoop.fs.s3a.commit.staging.TestStagingPartitionedTaskCommit [INFO] Tests run: 63, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 61.11 s - in org.apache.hadoop.fs.s3a.commit.staging.TestStagingCommitter [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 84.193 s - in org.apache.hadoop.fs.s3a.commit.staging.TestDirectoryCommitterScale [INFO] Running org.apache.hadoop.fs.s3a.commit.staging.integration.ITestStagingCommitProtocol [INFO] Running org.apache.hadoop.fs.s3a.commit.staging.integration.ITestDirectoryCommitProtocol [INFO] Running org.apache.hadoop.fs.s3a.commit.staging.integration.ITestPartitionedCommitProtocol [INFO] Running org.apache.hadoop.fs.s3a.commit.staging.integration.ITestStagingCommitProtocolFailure [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.901 s - in .hadoop.fs.s3a.commit.staging.integration.ITestStagingCommitProtocolFailure [INFO] Running org.apache.hadoop.fs.s3a.commit.magic.ITestMagicCommitProtocol [INFO] Running org.apache.hadoop.fs.s3a.commit.magic.ITestMagicCommitProtocolFailure [INFO] Running org.apache.hadoop.fs.s3a.commit.integration.ITestS3ACommitterMRJob [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.225 s - in org.apache.hadoop.fs.s3a.commit.magic.ITestMagicCommitProtocolFailure [INFO] Running org.apache.hadoop.fs.s3a.commit.ITestS3ACommitterFactory [INFO] Running org.apache.hadoop.fs.s3a.commit.ITestCommitOperations [INFO] Running org.apache.hadoop.fs.s3a.commit.ITestCommitOperationCost [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.05 s - in org.apache.hadoop.fs.s3a.commit.ITestS3ACommitterFactory [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 43.398 s - in org.apache.hadoop.fs.s3a.commit.ITestCommitOperationCost [INFO] Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 143.308 s - in org.apache.hadoop.fs.s3a.commit.ITestCommitOperations [INFO] Running org.apache.hadoop.fs.s3a.auth.ITestAssumedRoleCommitOperations [INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 218.466 s - in org.apache.hadoop.fs.s3a.commit.integration.ITestS3ACommitterMRJob [WARNING] Tests run: 18, Failures: 0, Errors: 0, Skipped: 18, Time elapsed: 62.036 s - in org.apache.hadoop.fs.s3a.auth.ITestAssumedRoleCommitOperations [WARNING] Tests run: 24, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 429.367 s - in .hadoop.fs.s3a.commit.staging.integration.ITestPartitionedCommitProtocol [INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 472.06 s - in .hadoop.fs.s3a.commit.staging.integration.ITestStagingCommitProtocol [INFO] Tests run: 25, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 498.078 s - in .hadoop.fs.s3a.commit.staging.integration.ITestDirectoryCommitProtocol [INFO] Tests run: 23, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 861.607 s - in org.apache.hadoop.fs.s3a.commit.magic.ITestMagicCommitProtocol [INFO] Running org.apache.hadoop.fs.s3a.commit.magic.ITestS3AHugeMagicCommits [WARNING] Tests run: 10, Failures: 0, Errors: 0, Skipped: 10, Time elapsed: 29.141 s - in org.apache.hadoop.fs.s3a.commit.magic.ITestS3AHugeMagicCommits [INFO] Running org.apache.hadoop.fs.s3a.commit.terasort.ITestTerasortOnS3A [WARNING] Tests run: 14, Failures: 0, Errors: 0, Skipped: 14, Time elapsed: 47.485 s - in org.apache.hadoop.fs.s3a.commit.terasort.ITestTerasortOnS3A

[steveloughran]

thanks. we don't need the whole trace, just region, maven args (-Dscale -Dprefetch ...) and whether any tests failed. if a test fails, look in jira to see if it's known

[subu-zd]

@shameersss1 @steveloughran Was this patch released as part of any 3.3.x hadoop-aws releases. I see that the JIRA references 3.3.9 but that's not yet released if I'm correct. Can you confirm?