Skip to content

[SPARK-29403][INFRA][R] Uses Arrow R 0.14.1 in AppVeyor for now #26041

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
HyukjinKwon wants to merge 6 commits into from
Closed

[SPARK-29403][INFRA][R] Uses Arrow R 0.14.1 in AppVeyor for now #26041

HyukjinKwon wants to merge 6 commits into from

Conversation

@HyukjinKwon
Copy link
Member

@HyukjinKwon HyukjinKwon commented Oct 7, 2019 •
edited
Loading

What changes were proposed in this pull request?

This PR proposes to use Arrow R 0.14.1 for now in AppVeyor to make tests passed.

Why are the changes needed?

To make build passed with Arrow. It doesn't work with setting ARROW_PRE_0_15_IPC_FORMAT to 1 to allow Arrow R 0.15 compatibility.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

AppVeyor

@SparkQA

This comment has been minimized.

Sign in to view
@HyukjinKwon
Copy link
Member Author

HyukjinKwon commented Oct 7, 2019

cc @BryanCutler FYI

@HyukjinKwon
Copy link
Member Author

HyukjinKwon commented Oct 7, 2019

@BryanCutler did Arrow change the format itself? bunch of PySpark tests with Arrow fail too.

@SparkQA
Copy link

SparkQA commented Oct 7, 2019

Test build #111837 has finished for PR 26041 at commit f30f4de.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon
Copy link
Member Author

HyukjinKwon commented Oct 7, 2019

Seems Arrow isn't released yet in Windows. The AppVeyor build here used Areow 0.14.1.1. I'll trigger the build again when Areow 0.15 is available on Windows.

@harpaj
Copy link

harpaj commented Oct 7, 2019

Might have the same cause as Pandas UDFs not working with Arrow 0.15, see https://issues.apache.org/jira/projects/SPARK/issues/SPARK-29367

@BryanCutler
Copy link
Member

BryanCutler commented Oct 7, 2019

@BryanCutler did Arrow change the format itself? bunch of PySpark tests with Arrow fail too.

Yes, there was a change but I have a patch ready that will be backwards compatible. I'll open a JIRA for the upgrade and we can go over the details.

@BryanCutler
Copy link
Member

BryanCutler commented Oct 7, 2019

I opened a PR documenting the change in #26045

@SparkQA
Copy link

SparkQA commented Oct 8, 2019

Test build #111868 has finished for PR 26041 at commit 34da71d.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon HyukjinKwon changed the title [DO-NOT-MERGE] Investigate Arrow 0.15 with R Arrow optimization [SPARK-29378][INFRA][R] Make AppVeyor's SparkR with Arrow tests compatible with Arrow R 0.15 Oct 8, 2019
@HyukjinKwon HyukjinKwon changed the title [SPARK-29378][INFRA][R] Make AppVeyor's SparkR with Arrow tests compatible with Arrow R 0.15 [WIP][SPARK-29378][INFRA][R] Make AppVeyor's SparkR with Arrow tests compatible with Arrow R 0.15 Oct 8, 2019
@HyukjinKwon
Copy link
Member Author

HyukjinKwon commented Oct 8, 2019

Hm, setting ARROW_PRE_0_15_IPC_FORMAT to 1 doesn't seem working in SparkR case. I tested this both in Windows (via AppVeyor) and Ubuntu

@SparkQA
Copy link

SparkQA commented Oct 8, 2019

Test build #111872 has finished for PR 26041 at commit bea530f.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Oct 8, 2019

Test build #111871 has finished for PR 26041 at commit f969c17.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon
Copy link
Member Author

HyukjinKwon commented Oct 9, 2019

I am going to set the Arrow R version to 0.14.1 for now explicitly to avoid to break build. The build is borken in the master due to this.

@HyukjinKwon HyukjinKwon changed the title [WIP][SPARK-29378][INFRA][R] Make AppVeyor's SparkR with Arrow tests compatible with Arrow R 0.15 [SPARK-29403][INFRA][R] Uses Arrow R 0.14.1 in AppVeyor for now Oct 9, 2019
@SparkQA
Copy link

SparkQA commented Oct 9, 2019

Test build #111927 has started for PR 26041 at commit e1a6807.

BryanCutler
BryanCutler reviewed Oct 9, 2019
View reviewed changes
appveyor.yml Outdated
environment:
NOT_CRAN: true
# See SPARK-29378.
ARROW_PRE_0_15_IPC_FORMAT: 1
Copy link
Member

@BryanCutler BryanCutler Oct 9, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does setting this here carry it through to the worker?

Copy link
Member Author

@HyukjinKwon HyukjinKwon Oct 10, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed it in the latest commit :-).

@HyukjinKwon
Copy link
Member Author

HyukjinKwon commented Oct 10, 2019

I am merging this to keep our AppVeyor build alive for now ..

atronchi pushed a commit to atronchi/spark that referenced this pull request Oct 23, 2019
@HyukjinKwon
[SPARK-29403][INFRA][R] Uses Arrow R 0.14.1 in AppVeyor for now
7fd7b40 
### What changes were proposed in this pull request?

This PR proposes to use Arrow R 0.14.1 for now in AppVeyor to make tests passed.

### Why are the changes needed?

To make build passed with Arrow. It doesn't work with setting `ARROW_PRE_0_15_IPC_FORMAT` to `1` to allow Arrow R 0.15 compatibility.

### Does this PR introduce any user-facing change?

No.

### How was this patch tested?

AppVeyor

Closes apache#26041 from HyukjinKwon/investigate.

Authored-by: HyukjinKwon <[email protected]>
Signed-off-by: HyukjinKwon <[email protected]>
@imback82 imback82 mentioned this pull request Oct 30, 2019
Closed
@dongjoon-hyun
Copy link
Member

dongjoon-hyun commented Nov 16, 2019

Hi, All.
After we upgrade Arrow 0.15.1, AppVeyor is dead again~ I'm revisiting this issue in the following PR.

@BryanCutler BryanCutler mentioned this pull request Nov 16, 2019
Closed
@HyukjinKwon HyukjinKwon deleted the investigate branch March 3, 2020 01:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@BryanCutler BryanCutler BryanCutler left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@HyukjinKwon @SparkQA @harpaj @BryanCutler @dongjoon-hyun