HADOOP-18107 Adding scale test for vectored reads for large file #4273

mukund-thakur · 2022-05-06T19:40:28Z

part of HADOOP-18103.

Description of PR

How was this patch tested?

Reran the changed tests.

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

part of HADOOP-18103.

steveloughran

made some minor suggestions

...mon-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/ContractTestUtils.java

steveloughran · 2022-05-11T09:37:10Z

...tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/AbstractSTestS3AHugeFiles.java


  @Test
-  public void test_045_VectoredIOHugeFile() throws Throwable {
+  public void test_045_VectoredIoHugeFile() throws Throwable {


capital IO is OK here; your choice

Yeah I thought that was the problem but don't know why checkstyle is crying
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/AbstractSTestS3AHugeFiles.java:459: public void test_045_vectoredIoHugeFile() throws Throwable {:15: Name 'test_045_vectoredIoHugeFile' must match pattern '^[a-z][a-zA-Z0-9]*$'. [MethodName]

the filename with the _ breaks checkstyle rules. maybe we should add a new rule which permits

test_[0-9]+_[a-zA-Z0-9]*$

in test code only?

...mon-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/ContractTestUtils.java

mukund-thakur · 2022-05-13T03:03:28Z

@steveloughran I think it is ready to be merged. Just the java docs.

steveloughran

LGTM, made some minor comments

...tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/AbstractSTestS3AHugeFiles.java

...mon-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/ContractTestUtils.java

steveloughran · 2022-05-24T20:59:03Z

+1 pending that move into a constant, just for clarity. thanks

(btw, suggest another rebase to get rid of those javadoc issues)

mukund-thakur · 2022-05-24T21:39:13Z

(btw, suggest another rebase to get rid of those javadoc issues)

Will do the rebase of feature branch in one go after the smaller patches are merged.

steveloughran

+1 pending the javadoc correction

steveloughran · 2022-05-30T13:19:20Z

...mon-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/ContractTestUtils.java

  public static final int DEFAULT_IO_CHUNK_MODULUS_SIZE = 128;

+  /**
+   * Timeout in seconds for vectored read


nit: add a .

hadoop-yetus · 2022-05-31T23:55:17Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 44s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ feature-vectored-io Compile Tests _
+0 🆗	mvndep	14m 32s		Maven dependency ordering for branch
+1 💚	mvninstall	25m 4s		feature-vectored-io passed
+1 💚	compile	23m 26s		feature-vectored-io passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	20m 55s		feature-vectored-io passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	4m 27s		feature-vectored-io passed
+1 💚	mvnsite	3m 46s		feature-vectored-io passed
-1 ❌	javadoc	1m 52s	/branch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt	hadoop-common in feature-vectored-io failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.
-1 ❌	javadoc	1m 21s	/branch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt	hadoop-aws in feature-vectored-io failed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.
+1 💚	spotbugs	5m 7s		feature-vectored-io passed
+1 💚	shadedclient	22m 30s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 26s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 49s		the patch passed
+1 💚	compile	22m 38s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	22m 38s		root-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 generated 0 new + 1822 unchanged - 2 fixed = 1822 total (was 1824)
+1 💚	compile	20m 45s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	20m 45s		root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~~20.04-b07 generated 0 new + 1701 unchanged - 2 fixed = 1701 total (was 1703)
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	4m 10s	/results-checkstyle-root.txt	root: The patch generated 1 new + 12 unchanged - 0 fixed = 13 total (was 12)
+1 💚	mvnsite	3m 42s		the patch passed
-1 ❌	javadoc	1m 43s	/patch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt	hadoop-common in the patch failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.
-1 ❌	javadoc	1m 21s	/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt	hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~~20.04-b07 generated 1 new + 36 unchanged - 9 fixed = 37 total (was 45)
+1 💚	spotbugs	5m 14s		the patch passed
+1 💚	shadedclient	22m 33s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	18m 26s		hadoop-common in the patch passed.
+1 💚	unit	3m 12s		hadoop-aws in the patch passed.
+1 💚	asflicense	1m 38s		The patch does not generate ASF License warnings.
		243m 28s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4273/9/artifact/out/Dockerfile
GITHUB PR	#4273
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 86b0ecfd23ff 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	feature-vectored-io / `c734d47`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4273/9/testReport/
Max. process+thread count	2235 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4273/9/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

steveloughran

+1

* HADOOP-18107 Adding scale test for vectored reads for large file part of HADOOP-18103.

part of HADOOP-18103. Contributed By: Mukund Thakur

…che#4273) part of HADOOP-18103. Contributed By: Mukund Thakur

HADOOP-18107 Adding scale test for vectored reads for large file

55886f5

part of HADOOP-18103.

mukund-thakur requested a review from steveloughran May 6, 2022 19:40

mukund-thakur added 2 commits May 9, 2022 14:08

Checkstyle

53d006f

Fixing gavadoc cuased by rebase

425eef4

steveloughran requested changes May 11, 2022

View reviewed changes

apache deleted a comment from hadoop-yetus May 11, 2022

mukund-thakur added 2 commits May 11, 2022 17:23

Steve's comments

84d1e5f

Tired of checkstyle and javadoc

6c1ac7e

mukund-thakur requested a review from steveloughran May 13, 2022 03:03

steveloughran reviewed May 16, 2022

View reviewed changes

apache deleted a comment from hadoop-yetus May 16, 2022

mukund-thakur requested a review from steveloughran May 24, 2022 20:26

mukund-thakur added 2 commits May 24, 2022 16:03

final review comments

4ab2e2f

Uncommenting a line

6c00e73

apache deleted a comment from hadoop-yetus May 25, 2022

checkstyle

ad3ceca

steveloughran requested changes May 30, 2022

View reviewed changes

Checkstyle

c734d47

mukund-thakur requested a review from steveloughran June 1, 2022 00:42

apache deleted a comment from hadoop-yetus Jun 1, 2022

steveloughran approved these changes Jun 1, 2022

View reviewed changes

mukund-thakur merged commit cbcb6a1 into apache:feature-vectored-io Jun 1, 2022

mukund-thakur added a commit that referenced this pull request Jun 15, 2022

HADOOP-18107 Adding scale test for vectored reads for large file (#4273)

9843cc0

* HADOOP-18107 Adding scale test for vectored reads for large file part of HADOOP-18103.

mukund-thakur added a commit that referenced this pull request Jun 21, 2022

HADOOP-18107 Adding scale test for vectored reads for large file (#4273)

0640790

part of HADOOP-18103. Contributed By: Mukund Thakur

asfgit pushed a commit that referenced this pull request Jun 22, 2022

HADOOP-18107 Adding scale test for vectored reads for large file (#4273)

1408dd8

part of HADOOP-18103. Contributed By: Mukund Thakur

mukund-thakur added a commit that referenced this pull request Jun 27, 2022

HADOOP-18107 Adding scale test for vectored reads for large file (#4273)

bb5a17b

part of HADOOP-18103. Contributed By: Mukund Thakur

HarshitGupta11 pushed a commit to HarshitGupta11/hadoop that referenced this pull request Nov 28, 2022

HADOOP-18107 Adding scale test for vectored reads for large file (apa…

ba337b2

…che#4273) part of HADOOP-18103. Contributed By: Mukund Thakur

HADOOP-18107 Adding scale test for vectored reads for large file #4273

HADOOP-18107 Adding scale test for vectored reads for large file #4273

Uh oh!

Conversation

mukund-thakur commented May 6, 2022

Description of PR

How was this patch tested?

For code changes:

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

steveloughran May 11, 2022

Choose a reason for hiding this comment

Uh oh!

mukund-thakur May 11, 2022

Choose a reason for hiding this comment

Uh oh!

steveloughran May 16, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mukund-thakur commented May 13, 2022

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

steveloughran commented May 24, 2022

Uh oh!

mukund-thakur commented May 24, 2022

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

steveloughran May 30, 2022

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented May 31, 2022

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants