Skip to content

[MINOR][SPARKR][DOC] Add a description for running unit tests in Windows #13217

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
HyukjinKwon wants to merge 5 commits into from
Closed

[MINOR][SPARKR][DOC] Add a description for running unit tests in Windows #13217

Changes from 4 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
8797ba3
Update WINDOWS.md so that users can run unit tests on Windows
HyukjinKwon May 20, 2016
ad68456
Correct indentation
HyukjinKwon May 20, 2016
acb9363
Add a missing parentheses
HyukjinKwon May 20, 2016
b62b631
Fix sentense
HyukjinKwon May 20, 2016
aa2839c
Address comments
HyukjinKwon May 21, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions R/WINDOWS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,3 +11,19 @@ include Rtools and R in `PATH`.
directory in Maven in `PATH`.
4. Set `MAVEN_OPTS` as described in [Building Spark](http://spark.apache.org/docs/latest/building-spark.html).
5. Open a command shell (`cmd`) in the Spark directory and run `mvn -DskipTests -Psparkr package`

## Unit tests

To run existing unit tests in SparkR on Windows, the following setps are required (the steps below suppose you are in Spark root directory)
Copy link
Member

@srowen srowen May 20, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Parenthetical is fine as a sentence by itself. "in the Spark root ..."

Copy link
Member Author

@HyukjinKwon HyukjinKwon May 20, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh. Thanks!


1. Set `HADOOP_HOME`.
2. Download `winutils.exe` and locate this in `$HADOOP_HOME/bin`.

It seems not requiring installing Hadoop but only this `winutils.exe`. It seems not included in Hadoop official binary releases so it should be built from source but it seems it is able to be downloaded from community (e.g. [steveloughran/winutils](https://github.com/steveloughran/winutils)).
Copy link
Member

@srowen srowen May 20, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CC @steveloughran for comment. I think the paragraph should start with "It is not included in the Hadoop binary releases, so .... However it is downloadable from, for example [...]"

Copy link
Member Author

@HyukjinKwon HyukjinKwon May 20, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you. I will wait for the comment and will fix.

Copy link
Contributor

@steveloughran steveloughran May 20, 2016 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wouldn't recommend putting it under the root of the project, as that only complicates the source tree and path cleanup; an adjacent directory works. And I think you may find that HADOOP.DLL is needed in places, as there are some JNI calls related to local file access and permissions/ACLs

At some point we (the Hadoop team) may start releasing the Windows binaries direct. It's only avoided as it complicates the release process somewhat, though if it encourages testing it can only be encouraged.

I'd suggest the following text:

To run the SparkR unit tests on Windows, the following steps are required —assuming you are in the Spark root directory and do not have Apache Hadoop installed already:

  1. cd ..
  2. mkdir hadoop
  3. Download the relevant Hadoop bin package from steveloughran/winutils. While these are not official ASF artifacts, they are built from the ASF release git hashes by a Hadoop PMC member on a dedicated Windows VM.
  4. Install the files into hadoop\bin; make sure that winutils.exe and hadoop.dll are present.
  5. Set the environment variable HADOOP_HOME to the full path to the newly created hadoop directory.
  6. For further reading, consult Windows Problems on the Hadoop wiki


3. Run unit-tests for SparkR by running below (you need to install the [testthat](http://cran.r-project.org/web/packages/testthat/index.html) package first):
Copy link
Member

@srowen srowen May 20, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"unit tests" and "by running the command below". Again parenthetical can be a sentence. This step is already documented in R docs though.

Copy link
Member Author

@HyukjinKwon HyukjinKwon May 20, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you. I will fix. I wrote it just in case becase the commends are a little bit different (I am not used to Windows and it took me a while to find out the equivalent commands).


```
R -e "install.packages('testthat', repos='http://cran.us.r-project.org')"
.\bin\spark-submit2.cmd --conf spark.hadoop.fs.defualt.name="file:///" R\pkg\tests\run-all.R
```