YARN-11195. Adding document to enable numa #4501
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
Contributor
There was a problem hiding this comment.
8192 does not look a right example value for cpus.
Contributor
There was a problem hiding this comment.
NodeManager instead of NameNode
Contributor
There was a problem hiding this comment.
NodeManager instead of namenode.
"grep the logs" could be modified to "grep the nodemanager log file using below command"
Contributor
There was a problem hiding this comment.
To check if container is assigned with NUMA - below can be used. And also grep the NodeManager log using below grep command
grep "NUMA node" | grep <container_id>
Contributor
There was a problem hiding this comment.
Can we also mention that NUMA support for YARN Container has to be enabled only if worker node machines has NUMA support.
Contributor
There was a problem hiding this comment.
Can we share the sample numa hardware output which shows node ids 0, 1 with memory and cpu configs from which user sets the below configs.
Contributor
There was a problem hiding this comment.
Can we describe why we set 8192 in this example.
Contributor
There was a problem hiding this comment.
Spark, Tez and other YARN Applications also need to set the container JVM Opts to leverage NUMA Support.
Contributor
|
Thanks @Samrat002 for the patch. Have given few comments. |
|
🎊 +1 overall
This message was automatically generated. |
Contributor
Author
|
@PrabhuJoseph updated with changes ! |
YARN-11195. Fix markdown path YARN-11195. Add optional numa balancing step YARN-11195. Fix indentation YARN-11195. Fix indentation YARN-11195. Numa Enabling document YARN-11195. Numa Enabling document
|
🎊 +1 overall
This message was automatically generated. |
Contributor
|
Thanks @Samrat002. Latest patch LGTM, +1. |
HarshitGupta11
pushed a commit
to HarshitGupta11/hadoop
that referenced
this pull request
Nov 28, 2022
Contributed by Samrat Deb.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of PR
This pr is a documentation steps to enable numa for the cluster running on instances like m5.24x large which internally has 2 chip .
Relevent JIRA :- YARN-11195
How was this patch tested?
This working steps has been tested in EMR cluster with 1 master node and 5 core nodes of instance types (m5.24xlarge)
Allignment
For code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?