Skip to content

Commit b5df92d

Browse files
felixcheungcmonkey
authored andcommitted
[SPARK-19324][SPARKR] Spark VJM stdout output is getting dropped in SparkR
## What changes were proposed in this pull request? This affects mostly running job from the driver in client mode when results are expected to be through stdout (which should be somewhat rare, but possible) Before: ``` > a <- as.DataFrame(cars) > b <- group_by(a, "dist") > c <- count(b) > sparkR.callJMethod(c$countjc, "explain", TRUE) NULL ``` After: ``` > a <- as.DataFrame(cars) > b <- group_by(a, "dist") > c <- count(b) > sparkR.callJMethod(c$countjc, "explain", TRUE) count#11L NULL ``` Now, `column.explain()` doesn't seem very useful (we can get more extensive output with `DataFrame.explain()`) but there are other more complex examples with calls of `println` in Scala/JVM side, that are getting dropped. ## How was this patch tested? manual Author: Felix Cheung <[email protected]> Closes apache#16670 from felixcheung/rjvmstdout.
1 parent 33eb50a commit b5df92d

File tree

2 files changed

+9
-4
lines changed

2 files changed

+9
-4
lines changed

R/pkg/R/utils.R

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -756,12 +756,17 @@ varargsToJProperties <- function(...) {
756756
props
757757
}
758758

759-
launchScript <- function(script, combinedArgs, capture = FALSE) {
759+
launchScript <- function(script, combinedArgs, wait = FALSE) {
760760
if (.Platform$OS.type == "windows") {
761761
scriptWithArgs <- paste(script, combinedArgs, sep = " ")
762-
shell(scriptWithArgs, translate = TRUE, wait = capture, intern = capture) # nolint
762+
# on Windows, intern = F seems to mean output to the console. (documentation on this is missing)
763+
shell(scriptWithArgs, translate = TRUE, wait = wait, intern = wait) # nolint
763764
} else {
764-
system2(script, combinedArgs, wait = capture, stdout = capture)
765+
# http://stat.ethz.ch/R-manual/R-devel/library/base/html/system2.html
766+
# stdout = F means discard output
767+
# stdout = "" means to its console (default)
768+
# Note that the console of this child process might not be the same as the running R process.
769+
system2(script, combinedArgs, stdout = "", wait = wait)
765770
}
766771
}
767772

R/pkg/inst/tests/testthat/test_Windows.R

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ test_that("sparkJars tag in SparkContext", {
2020
if (.Platform$OS.type != "windows") {
2121
skip("This test is only for Windows, skipped")
2222
}
23-
testOutput <- launchScript("ECHO", "a/b/c", capture = TRUE)
23+
testOutput <- launchScript("ECHO", "a/b/c", wait = TRUE)
2424
abcPath <- testOutput[1]
2525
expect_equal(abcPath, "a\\b\\c")
2626
})

0 commit comments

Comments
 (0)