HADOOP-18805. S3A prefetch tests to work with small files #5887

virajjasani · 2023-07-24T22:54:57Z

branch-3.3 backport of PR: #5851

virajjasani · 2023-07-24T23:15:15Z

Some relevant failures, will get to them:

[ERROR] testStatusProbesAfterClosingStream(org.apache.hadoop.fs.s3a.ITestS3APrefetchingInputStream)  Time elapsed: 0.34 s  <<< ERROR!
java.io.IOException: Unable to initialize filesystem s3a://{xyz}: java.lang.IllegalArgumentException: Value of fs.s3a.prefetch.block.size: 10240 is below the minimum value 8388608
	at org.apache.hadoop.fs.contract.AbstractBondedFSContract.init(AbstractBondedFSContract.java:76)
	at org.apache.hadoop.fs.contract.AbstractFSContractTestBase.setup(AbstractFSContractTestBase.java:187)
	at org.apache.hadoop.fs.s3a.AbstractS3ATestBase.setup(AbstractS3ATestBase.java:111)


[ERROR] testSeeksWithLruEviction[max-blocks-1](org.apache.hadoop.fs.s3a.ITestS3APrefetchingLruEviction)  Time elapsed: 0.338 s  <<< ERROR!
java.io.IOException: Unable to initialize filesystem s3a://{xyz}: java.lang.IllegalArgumentException: Value of fs.s3a.prefetch.block.size: 10240 is below the minimum value 8388608
	at org.apache.hadoop.fs.contract.AbstractBondedFSContract.init(AbstractBondedFSContract.java:76)
	at org.apache.hadoop.fs.contract.AbstractFSContractTestBase.setup(AbstractFSContractTestBase.java:187)
	at org.apache.hadoop.fs.s3a.AbstractS3ATestBase.setup(AbstractS3ATestBase.java:111)
	at org.apache.hadoop.fs.s3a.performance.AbstractS3ACostTest.setup(AbstractS3ACostTest.java:122)

hadoop-yetus · 2023-07-25T01:34:24Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	11m 4s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ branch-3.3 Compile Tests _
+1 💚	mvninstall	52m 57s		branch-3.3 passed
+1 💚	compile	0m 51s		branch-3.3 passed
+1 💚	checkstyle	0m 31s		branch-3.3 passed
+1 💚	mvnsite	0m 45s		branch-3.3 passed
+1 💚	javadoc	0m 35s		branch-3.3 passed
+1 💚	spotbugs	1m 12s		branch-3.3 passed
+1 💚	shadedclient	40m 28s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 39s		the patch passed
+1 💚	compile	0m 31s		the patch passed
+1 💚	javac	0m 31s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 20s		the patch passed
+1 💚	mvnsite	0m 37s		the patch passed
+1 💚	javadoc	0m 24s		the patch passed
+1 💚	spotbugs	1m 9s		the patch passed
+1 💚	shadedclient	39m 47s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 38s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 35s		The patch does not generate ASF License warnings.
		158m 21s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5887/1/artifact/out/Dockerfile
GITHUB PR	#5887
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 49e3a4da59cb 4.15.0-212-generic #223-Ubuntu SMP Tue May 23 13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / `9db8bd1`
Default Java	Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~18.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5887/1/testReport/
Max. process+thread count	621 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5887/1/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

virajjasani · 2023-07-25T01:37:08Z

@steveloughran we need HADOOP-18246 (commit: 22f6d55) on branch-3.3. i have tested by including that commit locally and now test results look good.

would you like to first directly merge it as separate commit to branch-3.3?

virajjasani · 2023-07-25T01:56:06Z

us-west-2:

$ mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale -Dprefetch

[INFO] ------------------------------------------------------------------------
[INFO] Total time:  30:52 min
[INFO] Finished at: 2023-07-24T18:55:36-07:00
[INFO] ------------------------------------------------------------------------

steveloughran · 2023-07-25T10:22:48Z

will do

steveloughran · 2023-07-25T11:33:42Z

ok, merged the dependency

…05-branch-3.3

virajjasani · 2023-07-25T18:40:35Z

rebased, re-testing

virajjasani · 2023-07-25T20:54:23Z

us-west-2:

$ mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale -Dprefetch

[INFO] ------------------------------------------------------------------------
[INFO] Total time:  40:12 min
[INFO] Finished at: 2023-07-25T12:20:03-07:00
[INFO] ------------------------------------------------------------------------

$ mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale

[INFO] ------------------------------------------------------------------------
[INFO] Total time:  44:16 min
[INFO] Finished at: 2023-07-25T13:18:57-07:00
[INFO] ------------------------------------------------------------------------

hadoop-yetus · 2023-07-25T21:09:21Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 49s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ branch-3.3 Compile Tests _
+1 💚	mvninstall	53m 18s		branch-3.3 passed
+1 💚	compile	0m 38s		branch-3.3 passed
+1 💚	checkstyle	0m 31s		branch-3.3 passed
+1 💚	mvnsite	0m 44s		branch-3.3 passed
+1 💚	javadoc	0m 35s		branch-3.3 passed
+1 💚	spotbugs	1m 14s		branch-3.3 passed
+1 💚	shadedclient	42m 5s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 40s		the patch passed
+1 💚	compile	0m 30s		the patch passed
+1 💚	javac	0m 30s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 19s		the patch passed
+1 💚	mvnsite	0m 35s		the patch passed
+1 💚	javadoc	0m 24s		the patch passed
+1 💚	spotbugs	1m 13s		the patch passed
+1 💚	shadedclient	39m 57s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 38s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 36s		The patch does not generate ASF License warnings.
		150m 2s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5887/2/artifact/out/Dockerfile
GITHUB PR	#5887
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux cc919073d6d9 4.15.0-212-generic #223-Ubuntu SMP Tue May 23 13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / `7a8fb11`
Default Java	Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~18.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5887/2/testReport/
Max. process+thread count	527 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5887/2/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

steveloughran · 2023-07-26T14:01:01Z

i did the merge by hand after cherrypicking prerequisites; closing this

HADOOP-18805. S3A prefetch tests to work with small files

9db8bd1

Merge branch 'branch-3.3' of github.com:apache/hadoop into HADOOP-188…

7a8fb11

…05-branch-3.3

steveloughran closed this Jul 26, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HADOOP-18805. S3A prefetch tests to work with small files #5887

HADOOP-18805. S3A prefetch tests to work with small files #5887

Uh oh!

virajjasani commented Jul 24, 2023

Uh oh!

virajjasani commented Jul 24, 2023

Uh oh!

hadoop-yetus commented Jul 25, 2023

Uh oh!

virajjasani commented Jul 25, 2023

Uh oh!

virajjasani commented Jul 25, 2023

Uh oh!

steveloughran commented Jul 25, 2023

Uh oh!

steveloughran commented Jul 25, 2023

Uh oh!

virajjasani commented Jul 25, 2023

Uh oh!

virajjasani commented Jul 25, 2023

Uh oh!

hadoop-yetus commented Jul 25, 2023

Uh oh!

steveloughran commented Jul 26, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

HADOOP-18805. S3A prefetch tests to work with small files #5887

HADOOP-18805. S3A prefetch tests to work with small files #5887

Uh oh!

Conversation

virajjasani commented Jul 24, 2023

Uh oh!

virajjasani commented Jul 24, 2023

Uh oh!

hadoop-yetus commented Jul 25, 2023

Uh oh!

virajjasani commented Jul 25, 2023

Uh oh!

virajjasani commented Jul 25, 2023

Uh oh!

steveloughran commented Jul 25, 2023

Uh oh!

steveloughran commented Jul 25, 2023

Uh oh!

virajjasani commented Jul 25, 2023

Uh oh!

virajjasani commented Jul 25, 2023

Uh oh!

hadoop-yetus commented Jul 25, 2023

Uh oh!

steveloughran commented Jul 26, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants