Skip to content

Assorted clean-up for Spark-on-YARN. #488

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
pwendell wants to merge 3 commits into from
Closed

Assorted clean-up for Spark-on-YARN. #488

pwendell wants to merge 3 commits into from
+18 −10

Conversation

@pwendell
Copy link
Contributor

@pwendell pwendell commented Apr 22, 2014

In particular when the HADOOP_CONF_DIR is not not specified.

@pwendell
Assorted clean-up for Spark-on-YARN.
17929cc 
In particular when the HADOOP_CONF_DIRS are not specified.
@AmplabJenkins
Copy link

AmplabJenkins commented Apr 22, 2014

Merged build triggered.

@AmplabJenkins
Copy link

AmplabJenkins commented Apr 22, 2014

Merged build started.

@pwendell pwendell mentioned this pull request Apr 22, 2014
Closed
andrewor14
andrewor14 reviewed Apr 22, 2014
View reviewed changes
conf/spark-env.sh.template
Copy link
Contributor

@andrewor14 andrewor14 Apr 22, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this be under "Options read in YARN client mode"?

Copy link
Contributor Author

@pwendell pwendell Apr 22, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added it there as well, but I'm going to leave it here because it's relevant also to yarn-cluster.

@AmplabJenkins
Copy link

AmplabJenkins commented Apr 22, 2014

Merged build triggered.

@AmplabJenkins
Copy link

AmplabJenkins commented Apr 22, 2014

Merged build started.

@AmplabJenkins
Copy link

AmplabJenkins commented Apr 22, 2014

Merged build finished.

@AmplabJenkins
Copy link

AmplabJenkins commented Apr 22, 2014

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14339/

@AmplabJenkins
Copy link

AmplabJenkins commented Apr 22, 2014

Merged build finished. All automated tests passed.

@AmplabJenkins
Copy link

AmplabJenkins commented Apr 22, 2014

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14341/

andrewor14
andrewor14 reviewed Apr 23, 2014
View reviewed changes
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala Outdated
Copy link
Contributor

@andrewor14 andrewor14 Apr 23, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

super small nit: I meant something more like if (!hasHadoopEnv && !testing) here, since the variable hasHadoopEnv should really not depend on whether we're testing Spark or not.

Copy link
Contributor Author

@pwendell pwendell Apr 23, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah that makes way more sense. Thanks

@hsaputra
Copy link
Contributor

hsaputra commented Apr 23, 2014

+1

Looks good.

@AmplabJenkins
Copy link

AmplabJenkins commented Apr 23, 2014

Merged build triggered.

@AmplabJenkins
Copy link

AmplabJenkins commented Apr 23, 2014

Merged build started.

@AmplabJenkins
Copy link

AmplabJenkins commented Apr 23, 2014

Merged build finished. All automated tests passed.

@AmplabJenkins
Copy link

AmplabJenkins commented Apr 23, 2014

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14351/

@pwendell
Copy link
Contributor Author

pwendell commented Apr 23, 2014

Okay merged

@asfgit asfgit closed this in 995fdc9 Apr 23, 2014
asfgit pushed a commit that referenced this pull request Apr 23, 2014
@pwendell
Assorted clean-up for Spark-on-YARN.
f764f47 
In particular when the HADOOP_CONF_DIR is not not specified.

Author: Patrick Wendell <[email protected]>

Closes #488 from pwendell/hadoop-cleanup and squashes the following commits:

fe95f13 [Patrick Wendell] Changes based on Andrew's feeback
18d09c1 [Patrick Wendell] Review comments from Andrew
17929cc [Patrick Wendell] Assorted clean-up for Spark-on-YARN.
(cherry picked from commit 995fdc9)

Signed-off-by: Patrick Wendell <[email protected]>
pdeyhim pushed a commit to pdeyhim/spark-1 that referenced this pull request Jun 25, 2014
@pwendell @pdeyhim
Assorted clean-up for Spark-on-YARN.
1712164 
In particular when the HADOOP_CONF_DIR is not not specified.

Author: Patrick Wendell <[email protected]>

Closes apache#488 from pwendell/hadoop-cleanup and squashes the following commits:

fe95f13 [Patrick Wendell] Changes based on Andrew's feeback
18d09c1 [Patrick Wendell] Review comments from Andrew
17929cc [Patrick Wendell] Assorted clean-up for Spark-on-YARN.
mccheah referenced this pull request in palantir/spark Sep 26, 2017
@foxish @mccheah
Improve the image building workflow (#488)
be394c6
markhamstra pushed a commit to markhamstra/spark that referenced this pull request Nov 7, 2017
@foxish
Improve the image building workflow (apache#488)
52fe7f5
bzhaoopenstack pushed a commit to bzhaoopenstack/spark that referenced this pull request Sep 11, 2019
@ZhengZhenyu
fix param pass (apache#488)
ad2df1b 
* fix param pass

fix param pass

* fix param pass

fix param pass

* fix param pass

fix param pass
turboFei pushed a commit to turboFei/spark that referenced this pull request Nov 6, 2025
@cloud-fan
[SPARK-46037][SQL] Correctness fix for Shuffled Hash Join build left …
9b4fd87 
…without codegen (apache#488)

[SPARK-46037][SQL] Correctness fix for Shuffled Hash Join build left without codegen

This is a re-submitting of apache#43938 to fix a join correctness bug caused by apache#41398 . Credits go to mcdull-zhang

correctness fix

Yes, the query result will be corrected.

new test

no

Closes apache#47905 from cloud-fan/join.

Authored-by: Wenchen Fan <[email protected]>

Signed-off-by: Wenchen Fan <[email protected]>
Co-authored-by: Wenchen Fan <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@pwendell @AmplabJenkins @hsaputra @andrewor14