HADOOP-17089: WASB: Update azure-storage-java SDK #2099

ThomasMarquardt · 2020-06-24T18:46:45Z

HADOOP-17089: WASB: Update azure-storage-java SDK

DETAILS: WASB depends on the Azure Storage Java SDK. There is a concurrency bug in the Azure Storage Java SDK that can cause the results of a list blobs operation to appear empty. This causes the Filesystem listStatus and similar APIs to return empty results. This has been seen in Spark work loads when jobs use more than one executor core.

See Azure/azure-storage-java#546 for details on the bug in the Azure Storage SDK.

TESTS: A new test was added to validate the fix. All tests are passing:

wasb:
mvn -T 1C -Dparallel-tests=wasb -Dscale -DtestsThreadCount=8 clean verify
Tests run: 248, Failures: 0, Errors: 0, Skipped: 11
Tests run: 651, Failures: 0, Errors: 0, Skipped: 65

abfs:
mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
Tests run: 64, Failures: 0, Errors: 0, Skipped: 0
Tests run: 437, Failures: 0, Errors: 0, Skipped: 33
Tests run: 206, Failures: 0, Errors: 0, Skipped: 24

hadoop-yetus · 2020-06-24T19:57:51Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 53s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
+1 💚	mvninstall	26m 16s	trunk passed
+1 💚	compile	0m 21s	trunk passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04
+1 💚	compile	0m 18s	trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚	mvnsite	0m 22s	trunk passed
+1 💚	shadedclient	45m 36s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 20s	trunk passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04
+1 💚	javadoc	0m 19s	trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 12s	the patch passed
+1 💚	compile	0m 12s	the patch passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04
+1 💚	javac	0m 12s	the patch passed
+1 💚	compile	0m 11s	the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚	javac	0m 11s	the patch passed
+1 💚	mvnsite	0m 15s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 2s	The patch has no ill-formed XML file.
+1 💚	shadedclient	17m 1s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 21s	the patch passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04
+1 💚	javadoc	0m 17s	the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
		_ Other Tests _
+1 💚	unit	0m 19s	hadoop-project in the patch passed.
+1 💚	asflicense	0m 32s	The patch does not generate ASF License warnings.
		69m 46s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2099/1/artifact/out/Dockerfile
GITHUB PR	#2099
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml
uname	Linux 50363da089ad 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `84110d8`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-2099/1/testReport/
Max. process+thread count	446 (vs. ulimit of 5500)
modules	C: hadoop-project U: hadoop-project
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-2099/1/console
versions	git=2.17.1 maven=3.6.0
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2020-06-25T01:50:36Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 33s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+0 🆗	mvndep	0m 27s	Maven dependency ordering for branch
+1 💚	mvninstall	19m 2s	trunk passed
+1 💚	compile	19m 20s	trunk passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04
+1 💚	compile	16m 54s	trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚	checkstyle	3m 17s	trunk passed
+1 💚	mvnsite	1m 28s	trunk passed
+1 💚	shadedclient	19m 21s	branch has no errors when building and testing our client artifacts.
-1 ❌	javadoc	0m 43s	hadoop-azure in trunk failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.
+1 💚	javadoc	1m 16s	trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+0 🆗	spotbugs	1m 5s	Used deprecated FindBugs config; considering switching to SpotBugs.
+0 🆗	findbugs	0m 36s	branch/hadoop-project no findbugs output file (findbugsXml.xml)
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 30s	Maven dependency ordering for patch
+1 💚	mvninstall	0m 42s	the patch passed
+1 💚	compile	18m 46s	the patch passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04
+1 💚	javac	18m 46s	the patch passed
+1 💚	compile	16m 51s	the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚	javac	16m 51s	the patch passed
-0 ⚠️	checkstyle	2m 39s	root: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚	mvnsite	1m 24s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 1s	The patch has no ill-formed XML file.
+1 💚	shadedclient	14m 6s	patch has no errors when building and testing our client artifacts.
-1 ❌	javadoc	0m 42s	hadoop-azure in the patch failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.
+1 💚	javadoc	1m 16s	the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+0 🆗	findbugs	0m 35s	hadoop-project has no data from findbugs
		_ Other Tests _
+1 💚	unit	0m 34s	hadoop-project in the patch passed.
+1 💚	unit	1m 37s	hadoop-azure in the patch passed.
+1 💚	asflicense	0m 55s	The patch does not generate ASF License warnings.
		147m 33s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2099/2/artifact/out/Dockerfile
GITHUB PR	#2099
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle
uname	Linux 75880192a261 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `84110d8`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-2099/2/artifact/out/branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-2099/2/artifact/out/diff-checkstyle-root.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-2099/2/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-2099/2/testReport/
Max. process+thread count	458 (vs. ulimit of 5500)
modules	C: hadoop-project hadoop-tools/hadoop-azure U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-2099/2/console
versions	git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

DadanielZ

Thanks for the awesome fix!
There is a minor checkstyle issue, other than that it looks good to me!

...zure/src/test/java/org/apache/hadoop/fs/azure/ITestNativeAzureFileSystemConcurrencyLive.java

Contributed by Thomas Marquardt DETAILS: WASB depends on the Azure Storage Java SDK. There is a concurrency bug in the Azure Storage Java SDK that can cause the results of a list blobs operation to appear empty. This causes the Filesystem listStatus and similar APIs to return empty results. This has been seen in Spark work loads when jobs use more than one executor core. See Azure/azure-storage-java#546 for details on the bug in the Azure Storage SDK. TESTS: A new test was added to validate the fix. All tests are passing: wasb: mvn -T 1C -Dparallel-tests=wasb -Dscale -DtestsThreadCount=8 clean verify Tests run: 248, Failures: 0, Errors: 0, Skipped: 11 Tests run: 651, Failures: 0, Errors: 0, Skipped: 65 abfs: mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify Tests run: 64, Failures: 0, Errors: 0, Skipped: 0 Tests run: 437, Failures: 0, Errors: 0, Skipped: 33 Tests run: 206, Failures: 0, Errors: 0, Skipped: 24

hadoop-yetus · 2020-06-25T05:02:13Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 32s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+0 🆗	mvndep	0m 26s	Maven dependency ordering for branch
+1 💚	mvninstall	19m 3s	trunk passed
+1 💚	compile	19m 24s	trunk passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04
+1 💚	compile	16m 49s	trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚	checkstyle	2m 38s	trunk passed
+1 💚	mvnsite	1m 26s	trunk passed
+1 💚	shadedclient	18m 42s	branch has no errors when building and testing our client artifacts.
-1 ❌	javadoc	0m 42s	hadoop-azure in trunk failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.
+1 💚	javadoc	1m 16s	trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+0 🆗	spotbugs	1m 5s	Used deprecated FindBugs config; considering switching to SpotBugs.
+0 🆗	findbugs	0m 36s	branch/hadoop-project no findbugs output file (findbugsXml.xml)
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 25s	Maven dependency ordering for patch
+1 💚	mvninstall	0m 41s	the patch passed
+1 💚	compile	18m 47s	the patch passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04
+1 💚	javac	18m 47s	the patch passed
+1 💚	compile	16m 51s	the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚	javac	16m 51s	the patch passed
+1 💚	checkstyle	2m 43s	the patch passed
+1 💚	mvnsite	1m 25s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 2s	The patch has no ill-formed XML file.
+1 💚	shadedclient	14m 8s	patch has no errors when building and testing our client artifacts.
-1 ❌	javadoc	0m 43s	hadoop-azure in the patch failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.
+1 💚	javadoc	1m 15s	the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+0 🆗	findbugs	0m 34s	hadoop-project has no data from findbugs
		_ Other Tests _
+1 💚	unit	0m 35s	hadoop-project in the patch passed.
+1 💚	unit	1m 37s	hadoop-azure in the patch passed.
+1 💚	asflicense	0m 54s	The patch does not generate ASF License warnings.
		146m 56s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2099/3/artifact/out/Dockerfile
GITHUB PR	#2099
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle
uname	Linux 01be23e81775 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `84110d8`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-2099/3/artifact/out/branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-2099/3/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-2099/3/testReport/
Max. process+thread count	449 (vs. ulimit of 5500)
modules	C: hadoop-project hadoop-tools/hadoop-azure U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-2099/3/console
versions	git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

ThomasMarquardt · 2020-06-25T05:52:42Z

The javadoc issue is tracked by HADOOP-17085.

ThomasMarquardt force-pushed the trunk branch from 19e5b54 to 1d45647 Compare June 24, 2020 23:21

DadanielZ requested changes Jun 25, 2020

View reviewed changes

...zure/src/test/java/org/apache/hadoop/fs/azure/ITestNativeAzureFileSystemConcurrencyLive.java Outdated Show resolved Hide resolved

ThomasMarquardt force-pushed the trunk branch from 1d45647 to 4b5b54c Compare June 25, 2020 02:34

asfgit merged commit 4b5b54c into apache:trunk Jun 25, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HADOOP-17089: WASB: Update azure-storage-java SDK #2099

HADOOP-17089: WASB: Update azure-storage-java SDK #2099

Uh oh!

ThomasMarquardt commented Jun 24, 2020 •

edited

Loading

Uh oh!

hadoop-yetus commented Jun 24, 2020

Uh oh!

hadoop-yetus commented Jun 25, 2020

Uh oh!

DadanielZ left a comment

Uh oh!

Uh oh!

hadoop-yetus commented Jun 25, 2020

Uh oh!

ThomasMarquardt commented Jun 25, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

HADOOP-17089: WASB: Update azure-storage-java SDK #2099

HADOOP-17089: WASB: Update azure-storage-java SDK #2099

Uh oh!

Conversation

ThomasMarquardt commented Jun 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hadoop-yetus commented Jun 24, 2020

Uh oh!

hadoop-yetus commented Jun 25, 2020

Uh oh!

DadanielZ left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hadoop-yetus commented Jun 25, 2020

Uh oh!

ThomasMarquardt commented Jun 25, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ThomasMarquardt commented Jun 24, 2020 •

edited

Loading