Skip to content

[SPARK-20704][SPARKR] change CRAN test to run single thread #17945

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
felixcheung wants to merge 6 commits into from
Closed

[SPARK-20704][SPARKR] change CRAN test to run single thread #17945

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
468cef1
change CRAN test to run single thread
felixcheung May 11, 2017
92e6bba
old api
felixcheung May 11, 2017
a718645
rename
felixcheung May 11, 2017
21bea08
lower case
felixcheung May 11, 2017
25a95a9
not show local in vignette
felixcheung May 11, 2017
5aa7fd4
remove os_type
felixcheung May 11, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions R/pkg/DESCRIPTION
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ Authors@R: c(person("Shivaram", "Venkataraman", role = c("aut", "cre"),
License: Apache License (== 2.0)
URL: http://www.apache.org/ http://spark.apache.org/
BugReports: http://spark.apache.org/contributing.html
OS_Type: unix
Copy link
Contributor

@shivaram shivaram May 11, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we do this in a separate PR ? It'll be good to track this as a separate JIRA / PR just to lookup why we did this. Also I'll hopefully hear from the winbuilder maintainer if this is something we can avoid doing.

Copy link
Member Author

@felixcheung felixcheung May 11, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

right, I can revert this before merging. I want to have a branch to test devtools install

Depends:
R (>= 3.0),
methods
Expand Down
2 changes: 1 addition & 1 deletion R/pkg/inst/tests/testthat/jarTest.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
#
library(SparkR)

sc <- sparkR.session()
sc <- sparkR.session(master = "local[1]")
Copy link
Contributor

@shivaram shivaram May 11, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

any specific reason these dont use sparkRMaster variable ?

Copy link
Member Author

@felixcheung felixcheung May 11, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's not accessible here - this is a separate R app

Copy link
Contributor

@shivaram shivaram May 11, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah got it.


helloTest <- SparkR:::callJStatic("sparkrtest.DummyClass",
"helloWorld",
Expand Down
2 changes: 1 addition & 1 deletion R/pkg/inst/tests/testthat/packageInAJarTest.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
library(SparkR)
library(sparkPackageTest)

sparkR.session()
sparkR.session(master = "local[1]")

run1 <- myfunc(5L)

Expand Down
2 changes: 1 addition & 1 deletion R/pkg/inst/tests/testthat/test_Serde.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@

context("SerDe functionality")

sparkSession <- sparkR.session(enableHiveSupport = FALSE)
sparkSession <- sparkR.session(master = sparkRMaster, enableHiveSupport = FALSE)
Copy link
Contributor

@shivaram shivaram May 11, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor nit - Is sparkRTestMaster is a better name ?


test_that("SerDe of primitive types", {
skip_on_cran()
Expand Down
2 changes: 1 addition & 1 deletion R/pkg/inst/tests/testthat/test_binaryFile.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
context("functions on binary files")

# JavaSparkContext handle
sparkSession <- sparkR.session(enableHiveSupport = FALSE)
sparkSession <- sparkR.session(master = sparkRMaster, enableHiveSupport = FALSE)
sc <- callJStatic("org.apache.spark.sql.api.r.SQLUtils", "getJavaSparkContext", sparkSession)

mockFile <- c("Spark is pretty.", "Spark is awesome.")
Expand Down
2 changes: 1 addition & 1 deletion R/pkg/inst/tests/testthat/test_binary_function.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
context("binary functions")

# JavaSparkContext handle
sparkSession <- sparkR.session(enableHiveSupport = FALSE)
sparkSession <- sparkR.session(master = sparkRMaster, enableHiveSupport = FALSE)
sc <- callJStatic("org.apache.spark.sql.api.r.SQLUtils", "getJavaSparkContext", sparkSession)

# Data
Expand Down
2 changes: 1 addition & 1 deletion R/pkg/inst/tests/testthat/test_broadcast.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
context("broadcast variables")

# JavaSparkContext handle
sparkSession <- sparkR.session(enableHiveSupport = FALSE)
sparkSession <- sparkR.session(master = sparkRMaster, enableHiveSupport = FALSE)
sc <- callJStatic("org.apache.spark.sql.api.r.SQLUtils", "getJavaSparkContext", sparkSession)

# Partitioned data
Expand Down
16 changes: 8 additions & 8 deletions R/pkg/inst/tests/testthat/test_context.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ test_that("repeatedly starting and stopping SparkR", {
skip_on_cran()

for (i in 1:4) {
sc <- suppressWarnings(sparkR.init())
sc <- suppressWarnings(sparkR.init(master = sparkRMaster))
rdd <- parallelize(sc, 1:20, 2L)
expect_equal(countRDD(rdd), 20)
suppressWarnings(sparkR.stop())
Expand All @@ -69,7 +69,7 @@ test_that("repeatedly starting and stopping SparkR", {

test_that("repeatedly starting and stopping SparkSession", {
for (i in 1:4) {
sparkR.session(enableHiveSupport = FALSE)
sparkR.session(master = sparkRMaster, enableHiveSupport = FALSE)
df <- createDataFrame(data.frame(dummy = 1:i))
expect_equal(count(df), i)
sparkR.session.stop()
Expand All @@ -79,12 +79,12 @@ test_that("repeatedly starting and stopping SparkSession", {
test_that("rdd GC across sparkR.stop", {
skip_on_cran()

sc <- sparkR.sparkContext() # sc should get id 0
sc <- sparkR.sparkContext(master = sparkRMaster) # sc should get id 0
rdd1 <- parallelize(sc, 1:20, 2L) # rdd1 should get id 1
rdd2 <- parallelize(sc, 1:10, 2L) # rdd2 should get id 2
sparkR.session.stop()

sc <- sparkR.sparkContext() # sc should get id 0 again
sc <- sparkR.sparkContext(master = sparkRMaster) # sc should get id 0 again

# GC rdd1 before creating rdd3 and rdd2 after
rm(rdd1)
Expand All @@ -104,7 +104,7 @@ test_that("rdd GC across sparkR.stop", {
test_that("job group functions can be called", {
skip_on_cran()

sc <- sparkR.sparkContext()
sc <- sparkR.sparkContext(master = sparkRMaster)
setJobGroup("groupId", "job description", TRUE)
cancelJobGroup("groupId")
clearJobGroup()
Expand All @@ -118,7 +118,7 @@ test_that("job group functions can be called", {
test_that("utility function can be called", {
skip_on_cran()

sparkR.sparkContext()
sparkR.sparkContext(master = sparkRMaster)
setLogLevel("ERROR")
sparkR.session.stop()
})
Expand Down Expand Up @@ -175,7 +175,7 @@ test_that("sparkJars sparkPackages as comma-separated strings", {
})

test_that("spark.lapply should perform simple transforms", {
sparkR.sparkContext()
sparkR.sparkContext(master = sparkRMaster)
doubled <- spark.lapply(1:10, function(x) { 2 * x })
expect_equal(doubled, as.list(2 * 1:10))
sparkR.session.stop()
Expand All @@ -184,7 +184,7 @@ test_that("spark.lapply should perform simple transforms", {
test_that("add and get file to be downloaded with Spark job on every node", {
skip_on_cran()

sparkR.sparkContext()
sparkR.sparkContext(master = sparkRMaster)
# Test add file.
path <- tempfile(pattern = "hello", fileext = ".txt")
filename <- basename(path)
Expand Down
2 changes: 1 addition & 1 deletion R/pkg/inst/tests/testthat/test_includePackage.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
context("include R packages")

# JavaSparkContext handle
sparkSession <- sparkR.session(enableHiveSupport = FALSE)
sparkSession <- sparkR.session(master = sparkRMaster, enableHiveSupport = FALSE)
sc <- callJStatic("org.apache.spark.sql.api.r.SQLUtils", "getJavaSparkContext", sparkSession)

# Partitioned data
Expand Down
2 changes: 1 addition & 1 deletion R/pkg/inst/tests/testthat/test_jvm_api.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@

context("JVM API")

sparkSession <- sparkR.session(enableHiveSupport = FALSE)
sparkSession <- sparkR.session(master = sparkRMaster, enableHiveSupport = FALSE)

test_that("Create and call methods on object", {
jarr <- sparkR.newJObject("java.util.ArrayList")
Expand Down
2 changes: 1 addition & 1 deletion R/pkg/inst/tests/testthat/test_mllib_classification.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ library(testthat)
context("MLlib classification algorithms, except for tree-based algorithms")

# Tests for MLlib classification algorithms in SparkR
sparkSession <- sparkR.session(enableHiveSupport = FALSE)
sparkSession <- sparkR.session(master = sparkRMaster, enableHiveSupport = FALSE)

absoluteSparkPath <- function(x) {
sparkHome <- sparkR.conf("spark.home")
Expand Down
2 changes: 1 addition & 1 deletion R/pkg/inst/tests/testthat/test_mllib_clustering.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ library(testthat)
context("MLlib clustering algorithms")

# Tests for MLlib clustering algorithms in SparkR
sparkSession <- sparkR.session(enableHiveSupport = FALSE)
sparkSession <- sparkR.session(master = sparkRMaster, enableHiveSupport = FALSE)

absoluteSparkPath <- function(x) {
sparkHome <- sparkR.conf("spark.home")
Expand Down
2 changes: 1 addition & 1 deletion R/pkg/inst/tests/testthat/test_mllib_fpm.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ library(testthat)
context("MLlib frequent pattern mining")

# Tests for MLlib frequent pattern mining algorithms in SparkR
sparkSession <- sparkR.session(enableHiveSupport = FALSE)
sparkSession <- sparkR.session(master = sparkRMaster, enableHiveSupport = FALSE)

test_that("spark.fpGrowth", {
data <- selectExpr(createDataFrame(data.frame(items = c(
Expand Down
2 changes: 1 addition & 1 deletion R/pkg/inst/tests/testthat/test_mllib_recommendation.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ library(testthat)
context("MLlib recommendation algorithms")

# Tests for MLlib recommendation algorithms in SparkR
sparkSession <- sparkR.session(enableHiveSupport = FALSE)
sparkSession <- sparkR.session(master = sparkRMaster, enableHiveSupport = FALSE)

test_that("spark.als", {
data <- list(list(0, 0, 4.0), list(0, 1, 2.0), list(1, 1, 3.0), list(1, 2, 4.0),
Expand Down
2 changes: 1 addition & 1 deletion R/pkg/inst/tests/testthat/test_mllib_regression.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ library(testthat)
context("MLlib regression algorithms, except for tree-based algorithms")

# Tests for MLlib regression algorithms in SparkR
sparkSession <- sparkR.session(enableHiveSupport = FALSE)
sparkSession <- sparkR.session(master = sparkRMaster, enableHiveSupport = FALSE)

test_that("formula of spark.glm", {
skip_on_cran()
Expand Down
2 changes: 1 addition & 1 deletion R/pkg/inst/tests/testthat/test_mllib_stat.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ library(testthat)
context("MLlib statistics algorithms")

# Tests for MLlib statistics algorithms in SparkR
sparkSession <- sparkR.session(enableHiveSupport = FALSE)
sparkSession <- sparkR.session(master = sparkRMaster, enableHiveSupport = FALSE)

test_that("spark.kstest", {
data <- data.frame(test = c(0.1, 0.15, 0.2, 0.3, 0.25, -1, -0.5))
Expand Down
2 changes: 1 addition & 1 deletion R/pkg/inst/tests/testthat/test_mllib_tree.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ library(testthat)
context("MLlib tree-based algorithms")

# Tests for MLlib tree-based algorithms in SparkR
sparkSession <- sparkR.session(enableHiveSupport = FALSE)
sparkSession <- sparkR.session(master = sparkRMaster, enableHiveSupport = FALSE)

absoluteSparkPath <- function(x) {
sparkHome <- sparkR.conf("spark.home")
Expand Down
2 changes: 1 addition & 1 deletion R/pkg/inst/tests/testthat/test_parallelize_collect.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ numPairs <- list(list(1, 1), list(1, 2), list(2, 2), list(2, 3))
strPairs <- list(list(strList, strList), list(strList, strList))

# JavaSparkContext handle
sparkSession <- sparkR.session(enableHiveSupport = FALSE)
sparkSession <- sparkR.session(master = sparkRMaster, enableHiveSupport = FALSE)
jsc <- callJStatic("org.apache.spark.sql.api.r.SQLUtils", "getJavaSparkContext", sparkSession)

# Tests
Expand Down
2 changes: 1 addition & 1 deletion R/pkg/inst/tests/testthat/test_rdd.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
context("basic RDD functions")

# JavaSparkContext handle
sparkSession <- sparkR.session(enableHiveSupport = FALSE)
sparkSession <- sparkR.session(master = sparkRMaster, enableHiveSupport = FALSE)
sc <- callJStatic("org.apache.spark.sql.api.r.SQLUtils", "getJavaSparkContext", sparkSession)

# Data
Expand Down
2 changes: 1 addition & 1 deletion R/pkg/inst/tests/testthat/test_shuffle.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
context("partitionBy, groupByKey, reduceByKey etc.")

# JavaSparkContext handle
sparkSession <- sparkR.session(enableHiveSupport = FALSE)
sparkSession <- sparkR.session(master = sparkRMaster, enableHiveSupport = FALSE)
sc <- callJStatic("org.apache.spark.sql.api.r.SQLUtils", "getJavaSparkContext", sparkSession)

# Data
Expand Down
2 changes: 1 addition & 1 deletion R/pkg/inst/tests/testthat/test_sparkSQL.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ unsetHiveContext <- function() {
# Tests for SparkSQL functions in SparkR

filesBefore <- list.files(path = sparkRDir, all.files = TRUE)
sparkSession <- sparkR.session()
sparkSession <- sparkR.session(master = sparkRMaster)
sc <- callJStatic("org.apache.spark.sql.api.r.SQLUtils", "getJavaSparkContext", sparkSession)

mockLines <- c("{\"name\":\"Michael\"}",
Expand Down
2 changes: 1 addition & 1 deletion R/pkg/inst/tests/testthat/test_streaming.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ context("Structured Streaming")

# Tests for Structured Streaming functions in SparkR

sparkSession <- sparkR.session(enableHiveSupport = FALSE)
sparkSession <- sparkR.session(master = sparkRMaster, enableHiveSupport = FALSE)

jsonSubDir <- file.path("sparkr-test", "json", "")
if (.Platform$OS.type == "windows") {
Expand Down
2 changes: 1 addition & 1 deletion R/pkg/inst/tests/testthat/test_take.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ strList <- list("Dexter Morgan: Blood. Sometimes it sets my teeth on edge, ",
"raising me. But they're both dead now. I didn't kill them. Honest.")

# JavaSparkContext handle
sparkSession <- sparkR.session(enableHiveSupport = FALSE)
sparkSession <- sparkR.session(master = sparkRMaster, enableHiveSupport = FALSE)
sc <- callJStatic("org.apache.spark.sql.api.r.SQLUtils", "getJavaSparkContext", sparkSession)

test_that("take() gives back the original elements in correct count and order", {
Expand Down
2 changes: 1 addition & 1 deletion R/pkg/inst/tests/testthat/test_textFile.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
context("the textFile() function")

# JavaSparkContext handle
sparkSession <- sparkR.session(enableHiveSupport = FALSE)
sparkSession <- sparkR.session(master = sparkRMaster, enableHiveSupport = FALSE)
sc <- callJStatic("org.apache.spark.sql.api.r.SQLUtils", "getJavaSparkContext", sparkSession)

mockFile <- c("Spark is pretty.", "Spark is awesome.")
Expand Down
2 changes: 1 addition & 1 deletion R/pkg/inst/tests/testthat/test_utils.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
context("functions in utils.R")

# JavaSparkContext handle
sparkSession <- sparkR.session(enableHiveSupport = FALSE)
sparkSession <- sparkR.session(master = sparkRMaster, enableHiveSupport = FALSE)
sc <- callJStatic("org.apache.spark.sql.api.r.SQLUtils", "getJavaSparkContext", sparkSession)

test_that("convertJListToRList() gives back (deserializes) the original JLists
Expand Down
5 changes: 5 additions & 0 deletions R/pkg/tests/run-all.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,4 +31,9 @@ sparkRWhitelistSQLDirs <- c("spark-warehouse", "metastore_db")
invisible(lapply(sparkRWhitelistSQLDirs,
function(x) { unlink(file.path(sparkRDir, x), recursive = TRUE, force = TRUE)}))

sparkRMaster <- "local[1]"
if (identical(Sys.getenv("NOT_CRAN"), "true")) {
sparkRMaster <- ""
}

test_package("SparkR")
2 changes: 1 addition & 1 deletion R/pkg/vignettes/sparkr-vignettes.Rmd
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ We use default settings in which it runs in local mode. It auto downloads Spark
install.spark()
```
```{r, message=FALSE, results="hide"}
sparkR.session()
sparkR.session(master = "local[1]")
```

The operations in SparkR are centered around an R class called `SparkDataFrame`. It is a distributed collection of data organized into named columns, which is conceptually equivalent to a table in a relational database or a data frame in R, but with richer optimizations under the hood.
Expand Down