YARN-11172. Fix TestClientRMTokens#testDelegationToken introduced by HDFS-16563. #4408

zhengchenyu · 2022-06-06T12:01:46Z

Description of PR

UT fail after [HDFS-16563], other yarn PR is blocked.

[ERROR] testDelegationToken(org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens)  Time elapsed: 17.379 s  <<< FAILURE!
java.lang.AssertionError
	at org.junit.Assert.fail(Assert.java:87)
	at org.junit.Assert.assertTrue(Assert.java:42)
	at org.junit.Assert.assertTrue(Assert.java:53)
	at org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens.testDelegationToken(TestClientRMTokens.java:207)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

hadoop-yetus · 2022-06-06T17:40:20Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 40s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 27s		Maven dependency ordering for branch
+1 💚	mvninstall	24m 43s		trunk passed
+1 💚	compile	22m 47s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	20m 32s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	4m 23s		trunk passed
+1 💚	mvnsite	4m 2s		trunk passed
+1 💚	javadoc	3m 22s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 55s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	5m 53s		trunk passed
+1 💚	shadedclient	23m 2s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 31s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 6s		the patch passed
+1 💚	compile	22m 9s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	22m 9s		the patch passed
+1 💚	compile	20m 39s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	20m 39s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	4m 8s		the patch passed
+1 💚	mvnsite	4m 0s		the patch passed
+1 💚	javadoc	3m 14s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 46s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	5m 53s		the patch passed
+1 💚	shadedclient	23m 11s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	18m 30s		hadoop-common in the patch passed.
+1 💚	unit	97m 46s		hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	1m 37s		The patch does not generate ASF License warnings.
		337m 11s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4408/1/artifact/out/Dockerfile
GITHUB PR	#4408
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 1315f1331a20 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `fe51f95`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4408/1/testReport/
Max. process+thread count	1266 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4408/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

slfan1989 · 2022-06-07T00:32:22Z

But it is recommended to write the title clearly.

assertTrue(e.getMessage().contains("Token  has expired"));

use has expired may be better and remove extra spaces.

zhengchenyu · 2022-06-07T01:41:29Z

@Hexiaoqiao @steveloughran Can you please review the PR? Some unit test fail after HDFS-16563.

steveloughran · 2022-06-09T12:22:02Z

LambdaTestUtils.intercept is the way to test this; its our reimplementation of ScalaTest intercept. if the exception class or message is wrong, the exception is rethrown.

which would you prefer in jenkins log. a message saying "an assert true failed" or "here is the exception which was raised with all its stack"?

zhengchenyu · 2022-06-09T13:48:41Z

LambdaTestUtils.intercept is the way to test this; its our reimplementation of ScalaTest intercept. if the exception class or message is wrong, the exception is rethrown.

which would you prefer in jenkins log. a message saying "an assert true failed" or "here is the exception which was raised with all its stack"?

@steveloughran

other PR link:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/5/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt

Fail log are:

INFO] 
[ERROR] Errors: 
[ERROR] org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens.testDelegationToken(org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens)
[ERROR]   Run 1: TestClientRMTokens.testDelegationToken:207
[ERROR]   Run 2: TestClientRMTokens.testDelegationToken:138 禄 YarnRuntime java.net.BindExceptio...
[ERROR]   Run 3: TestClientRMTokens.testDelegationToken:138 禄 YarnRuntime java.net.BindExceptio...
[INFO] 
[INFO] 
[ERROR] Tests run: 3322, Failures: 0, Errors: 1, Skipped: 9
[INFO] 
[ERROR] There are test failures.

Exception message changed by HDFS-16563. But this unit test just check the message.

zhengchenyu · 2022-06-09T14:20:59Z

LambdaTestUtils.intercept is the way to test this; its our reimplementation of ScalaTest intercept. if the exception class or message is wrong, the exception is rethrown.

which would you prefer in jenkins log. a message saying "an assert true failed" or "here is the exception which was raised with all its stack"?

I will fix the code like

      final ApplicationClientProtocol finalClientRMWithDT = clientRMWithDT;
      final GetNewApplicationRequest finalRequest = request;
      LambdaTestUtils.intercept(InvalidToken.class, "Token  has expired",
          () -> finalClientRMWithDT.getNewApplication(finalRequest));

Should I fix all code in this style? Because I found many code in TestClientRMTokens need to fix in this way.

steveloughran · 2022-06-09T15:11:06Z

just do the one which is broken. yes, the rest is out of date, but it would make for a larger patch, harder to cherrypick into branch-3.3 etc.

with intercept() if the wrong exception type or message is found, the exception is thrown again, so we understand what is there.

…HDFS-16563.

steveloughran · 2022-06-09T12:19:02Z

...emanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMTokens.java

      } catch (Exception e) {
        assertEquals(InvalidToken.class.getName(), e.getClass().getName());
-        assertTrue(e.getMessage().contains("is expired"));
+        assertTrue(e.getMessage().contains("Token  has expired"));


this is pretty old code; we would never write new tests this way.
as well as being complicated, if a different exception is raised we want that stack trace

best to clean up

replace the whole section from L202 with a call of

LambdaTestUtils.intercept(InvalidToken.class, "has expired", () -> clientRMWithDT.getNewApplication(request));

look inside the intercept call to see what it does better.

I have fix this code according to your advice. Maybe it's outdated code, I have rebase it.

...emanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMTokens.java

zhengchenyu · 2022-06-13T02:53:29Z

@steveloughran Can you please review it again and merge to trunk? Some PR is blocked by this.

steveloughran

+1, merging

…DFS-16563. (#4408) Regression caused by HDFS-16563; the hdfs exception text was changed, but because it was a YARN test doing the check, Yetus didn't notice. Contributed by zhengchenyu

…HDFS-16563. (apache#4408) Regression caused by HDFS-16563; the hdfs exception text was changed, but because it was a YARN test doing the check, Yetus didn't notice. Contributed by zhengchenyu

zhengchenyu changed the title ~~YARN-11172. Fix testDelegationToken~~ YARN-11172. Fix TestClientRMTokens#testDelegationToken introduced by HDFS-16563. Jun 7, 2022

zhengchenyu closed this Jun 10, 2022

zhengchenyu deleted the YARN-11172 branch June 10, 2022 00:51

YARN-11172. Fix TestClientRMTokens#testDelegationToken introduced by …

235be55

…HDFS-16563.

zhengchenyu reopened this Jun 10, 2022

zhengchenyu closed this Jun 10, 2022

zhengchenyu reopened this Jun 10, 2022

zhengchenyu force-pushed the YARN-11172 branch from 869c6e5 to 235be55 Compare June 10, 2022 00:58

trigger ci

2ab32a9

steveloughran reviewed Jun 12, 2022

View reviewed changes

steveloughran approved these changes Jun 17, 2022

View reviewed changes

apache deleted a comment from hadoop-yetus Jun 17, 2022

steveloughran merged commit 80446dc into apache:trunk Jun 17, 2022

slfan1989 mentioned this pull request Jun 18, 2022

HADOOP-18284. Remove Unnecessary semicolon ';' #4422

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

YARN-11172. Fix TestClientRMTokens#testDelegationToken introduced by HDFS-16563. #4408

YARN-11172. Fix TestClientRMTokens#testDelegationToken introduced by HDFS-16563. #4408

Uh oh!

zhengchenyu commented Jun 6, 2022

Uh oh!

hadoop-yetus commented Jun 6, 2022

Uh oh!

slfan1989 commented Jun 7, 2022 •

edited

Loading

Uh oh!

zhengchenyu commented Jun 7, 2022

Uh oh!

steveloughran commented Jun 9, 2022

Uh oh!

zhengchenyu commented Jun 9, 2022 •

edited

Loading

Uh oh!

zhengchenyu commented Jun 9, 2022

Uh oh!

steveloughran commented Jun 9, 2022

Uh oh!

steveloughran Jun 9, 2022

Uh oh!

zhengchenyu Jun 12, 2022

Uh oh!

Uh oh!

zhengchenyu commented Jun 13, 2022

Uh oh!

steveloughran left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

YARN-11172. Fix TestClientRMTokens#testDelegationToken introduced by HDFS-16563. #4408

YARN-11172. Fix TestClientRMTokens#testDelegationToken introduced by HDFS-16563. #4408

Uh oh!

Conversation

zhengchenyu commented Jun 6, 2022

Description of PR

Uh oh!

hadoop-yetus commented Jun 6, 2022

Uh oh!

slfan1989 commented Jun 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zhengchenyu commented Jun 7, 2022

Uh oh!

steveloughran commented Jun 9, 2022

Uh oh!

zhengchenyu commented Jun 9, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zhengchenyu commented Jun 9, 2022

Uh oh!

steveloughran commented Jun 9, 2022

Uh oh!

steveloughran Jun 9, 2022

Choose a reason for hiding this comment

Uh oh!

zhengchenyu Jun 12, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

zhengchenyu commented Jun 13, 2022

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

slfan1989 commented Jun 7, 2022 •

edited

Loading

zhengchenyu commented Jun 9, 2022 •

edited

Loading