TEZ-4274: Log Credential Tokens #94

belugabehr · 2021-01-28T16:58:29Z

No description provided.

jteagles · 2021-01-28T17:07:07Z

Was accidentally dropped as part of TEZ-2176. MapReduce logs credentials with this message.

""Executing with tokens: {}", credentials.getAllTokens()"

I wonder if we want to Update the message to better reflect purpose.

belugabehr · 2021-01-28T18:05:08Z

What message would that be? Happy to update it.

jteagles · 2021-01-28T20:05:56Z

I wasn't clear in my last reply. MAPREDUCE-6971 simplified the way logging is done. In Tez we prefer to log credentials only in debug, but the concept is the same. TezChild was modeled after YarnChild, so it may make sense to adopt similar change. As show in the link below.
apache/hadoop@453d48b#diff-9191ed128249f1e8510e92cc7c1fa1160d1743f7ffebe82dd5dab0b2cb38ad79L103

belugabehr · 2021-07-27T16:06:00Z

@jteagles Updated as requested. Thanks!

hadoop-yetus · 2021-07-27T18:00:50Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	19m 57s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+1 💚	mvninstall	14m 49s	master passed
+1 💚	compile	0m 30s	master passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	0m 29s	master passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	1m 2s	master passed
+1 💚	javadoc	0m 38s	master passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 24s	master passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+0 🆗	spotbugs	1m 16s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	1m 14s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 19s	the patch passed
+1 💚	compile	0m 20s	the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	0m 20s	the patch passed
+1 💚	compile	0m 19s	the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	0m 19s	the patch passed
+1 💚	checkstyle	0m 11s	tez-runtime-internals: The patch generated 0 new + 8 unchanged - 1 fixed = 8 total (was 9)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	javadoc	0m 18s	the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 16s	the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	findbugs	0m 53s	the patch passed
		_ Other Tests _
+1 💚	unit	0m 45s	tez-runtime-internals in the patch passed.
+1 💚	asflicense	0m 15s	The patch does not generate ASF License warnings.
		43m 33s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-94/2/artifact/out/Dockerfile
GITHUB PR	#94
JIRA Issue	TEZ-4274
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
uname	Linux b093570f9306 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / `464d86d`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-94/2/testReport/
Max. process+thread count	90 (vs. ulimit of 5500)
modules	C: tez-runtime-internals U: tez-runtime-internals
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-94/2/console
versions	git=2.25.1 maven=3.6.3 findbugs=3.0.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

abstractdog · 2021-07-28T06:47:40Z

nice catch!

credentials are already logged in TezChild constructor by:
https://github.com/apache/tez/blob/master/tez-runtime-internals/src/main/java/org/apache/tez/runtime/task/TezChild.java#L203

    TezCommonUtils.logCredentials(LOG, credentials, "tezChildInit");

wouldn't this be enough? not 100% in line with hadoop, but adds a useful small context message

hadoop-yetus · 2021-07-29T09:40:52Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	12m 45s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+1 💚	mvninstall	12m 53s	master passed
+1 💚	compile	0m 29s	master passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	0m 29s	master passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	1m 0s	master passed
+1 💚	javadoc	0m 40s	master passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 27s	master passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+0 🆗	spotbugs	1m 11s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	1m 8s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 16s	the patch passed
+1 💚	compile	0m 16s	the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	0m 16s	the patch passed
+1 💚	compile	0m 14s	the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	0m 14s	the patch passed
+1 💚	checkstyle	0m 9s	tez-runtime-internals: The patch generated 0 new + 8 unchanged - 1 fixed = 8 total (was 9)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	javadoc	0m 14s	the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 13s	the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	findbugs	0m 37s	the patch passed
		_ Other Tests _
+1 💚	unit	0m 37s	tez-runtime-internals in the patch passed.
+1 💚	asflicense	0m 14s	The patch does not generate ASF License warnings.
		33m 34s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-94/3/artifact/out/Dockerfile
GITHUB PR	#94
JIRA Issue	TEZ-4274
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
uname	Linux 4802f1882e82 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / `464d86d`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-94/3/testReport/
Max. process+thread count	102 (vs. ulimit of 5500)
modules	C: tez-runtime-internals U: tez-runtime-internals
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-94/3/console
versions	git=2.25.1 maven=3.6.3 findbugs=3.0.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

tez-yetus · 2021-08-09T20:01:46Z

Thanks, @abstractdog. I totally missed this change. @belugabehr is the current credential logging enough for your needs?

jteagles · 2021-08-09T20:44:09Z

Sorry, the above message was from me.

abstractdog · 2022-11-17T10:40:28Z

closing this as housekeeping, please reopen if needed

TEZ-4274: Log Credential Tokens

bc1bfac

Update logging to match Hadoop proper

af21999

abstractdog closed this Nov 17, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

TEZ-4274: Log Credential Tokens #94

TEZ-4274: Log Credential Tokens #94

Uh oh!

belugabehr commented Jan 28, 2021

Uh oh!

jteagles commented Jan 28, 2021

Uh oh!

belugabehr commented Jan 28, 2021

Uh oh!

jteagles commented Jan 28, 2021

Uh oh!

belugabehr commented Jul 27, 2021

Uh oh!

hadoop-yetus commented Jul 27, 2021

Uh oh!

abstractdog commented Jul 28, 2021 •

edited

Loading

Uh oh!

hadoop-yetus commented Jul 29, 2021

Uh oh!

tez-yetus commented Aug 9, 2021

Uh oh!

jteagles commented Aug 9, 2021

Uh oh!

abstractdog commented Nov 17, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

TEZ-4274: Log Credential Tokens #94

TEZ-4274: Log Credential Tokens #94

Uh oh!

Conversation

belugabehr commented Jan 28, 2021

Uh oh!

jteagles commented Jan 28, 2021

Uh oh!

belugabehr commented Jan 28, 2021

Uh oh!

jteagles commented Jan 28, 2021

Uh oh!

belugabehr commented Jul 27, 2021

Uh oh!

hadoop-yetus commented Jul 27, 2021

Uh oh!

abstractdog commented Jul 28, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hadoop-yetus commented Jul 29, 2021

Uh oh!

tez-yetus commented Aug 9, 2021

Uh oh!

jteagles commented Aug 9, 2021

Uh oh!

abstractdog commented Nov 17, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

abstractdog commented Jul 28, 2021 •

edited

Loading