[BACKPORT-2.1][SPARKR][DOCS] update R API doc for subset/extract#16748

Closed

felixcheung wants to merge 484 commits intoapache:masterfrom

felixcheung:rsubsetdocbackport

This pull request is big! We're only showing the most recent 250 commits

Commits on Nov 30, 2016

[SPARK-18617][CORE][STREAMING] Close "kryo auto pick" feature for Spark Streaming

uncleGen
authored and
rxin
committed
[SPARK-18366][PYSPARK][ML] Add handleInvalid to Pyspark for QuantileDiscretizer and Bucketizer

techaddict
authored and
Nick Pentreath
committed
[SPARK-18612][MLLIB] Delete broadcasted variable in LBFGS CostFun

Anthony Truchet
authored and
srowen
committed
[SPARK-17897][SQL] Fixed IsNotNull Constraint Inference Rule

gatorsmile
authored and
cloud-fan
committed
[SPARK-18220][SQL] read Hive orc table with varchar column should not fail

cloud-fan
authored and
rxin
committed
[SPARK][EXAMPLE] Added missing semicolon in quick-start-guide example

manishAtGit
authored and
andrewor14
committed
[SPARK-18640] Add synchronization to TaskScheduler.runningTasksByExecutors

JoshRosen
authored and
andrewor14
committed
[SPARK-18318][ML] ML, Graph 2.1 QA: API: New Scala APIs, docs

yanboliang
authored and
jkbradley
committed
[SPARK-18251][SQL] the type of Dataset can't be Option of non-flat type

cloud-fan
authored and
liancheng
committed
[SPARK-18546][CORE] Fix merging shuffle spills when using encryption.
Marcelo Vanzin
committed

Commits on Dec 1, 2016

Commits on Dec 2, 2016

Commits on Dec 4, 2016

[SPARK-18081][ML][DOCS] Add user guide for Locality Sensitive Hashing(LSH)

Yunni
authored and
jkbradley
committed
[SPARK-18091][SQL] Deep if expressions cause Generated SpecificUnsafeProjection code to exceed JVM code size limit

Kapil Singh
authored and
cloud-fan
committed
[SPARK-18661][SQL] Creating a partitioned datasource table should not scan all files for table

ericl
authored and
cloud-fan
committed

Commits on Dec 5, 2016

Commits on Dec 6, 2016

[SPARK-18634][PYSPARK][SQL] Corruption and Correctness issues with exploding Python UDFs

viirya
authored and
hvanhovell
committed
[SPARK-18729][SS] Move DataFrame.collect out of synchronized block in MemorySink

zsxwing
authored and
tdas
committed
[SPARK-18657][SPARK-18668] Make StreamingQuery.id persists across restart and not auto-generate StreamingQuery.name
tdas
committed
[SPARK-18722][SS] Move no data rate limit from StreamExecution to ProgressReporter

zsxwing
authored and
tdas
committed
[SPARK-18572][SQL] Add a method `listPartitionNames` to `ExternalCatalog`

Michael Allman
authored and
cloud-fan
committed
[SPARK-18721][SS] Fix ForeachSink with watermark + append

zsxwing
authored and
tdas
committed
[SPARK-18634][SQL][TRIVIAL] Touch-up Generate
hvanhovell
committed
[SPARK-18714][SQL] Add a simple time function to SparkSession

rxin
authored and
hvanhovell
committed
[SPARK-18671][SS][TEST] Added tests to ensure stability of that all Structured Streaming log formats

tdas
authored and
zsxwing
committed
[SPARK-18652][PYTHON] Include the example data and third-party licenses in pyspark package.

lins05
authored and
srowen
committed

Commits on Dec 7, 2016

[SPARK-18734][SS] Represent timestamp in StreamingQueryProgress as formatted string instead of millis

tdas
authored and
zsxwing
committed
[SPARK-18671][SS][TEST-MAVEN] Follow up PR to fix test for Maven
tdas
committed
[SPARK-18686][SPARKR][ML] Several cleanup and improvements for spark.logit.
yanboliang
committed
[SPARK-18701][ML] Fix Poisson GLM failure due to wrong initialization

actuaryzhang
authored and
srowen
committed
[SPARK-18678][ML] Skewed reservoir sampling in SamplingUtils
srowen
committed
[SPARK-18208][SHUFFLE] Executor OOM due to a growing LongArray in BytesToBytesMap

Jie Xiong
authored and
hvanhovell
committed
[SPARK-17760][SQL] AnalysisException with dataframe pivot when groupBy column is not attribute

aray
authored and
hvanhovell
committed
[SPARK-18764][CORE] Add a warning log when skipping a corrupted file
zsxwing
committed
[SPARK-18762][WEBUI] Web UI should be http:4040 instead of https:4040

sarutak
authored and
Marcelo Vanzin
committed
[SPARK-18588][TESTS] Fix flaky test: KafkaSourceStressForDontFailOnDataLossSuite

zsxwing
authored and
tdas
committed
[SPARK-18754][SS] Rename recentProgresses to recentProgress

marmbrus
authored and
tdas
committed

Commits on Dec 8, 2016

Commits on Dec 9, 2016

[SPARK-18776][SS] Make Offset for FileStreamSource corrected formatted in json
tdas
committed
[SPARK-18774][CORE][SQL] Ignore non-existing files when ignoreCorruptFiles is enabled (branch 2.1)

zsxwing
authored and
rxin
committed
[SPARKR][PYSPARK] Fix R source package name to match Spark version. Remove pip tar.gz from distribution
shivaram
committed
[SPARK-18349][SPARKR] Update R API documentation on ml model summary

wangmiao1981
authored and
Felix Cheung
committed
Copy the SparkR source package with LFTP
shivaram
committed
Copy pyspark and SparkR packages to latest release dir too

felixcheung
authored and
shivaram
committed
[SPARK-18637][SQL] Stateful UDF should be considered as nondeterministic

Zhan Zhang
authored and
cloud-fan
committed
[MINOR][CORE][SQL][DOCS] Typo fixes

jaceklaskowski
authored and
srowen
committed
[SPARK-17822][R] Make JVMObjectTracker a member variable of RBackend
mengxr
committed
[MINOR][SPARKR] Fix SparkR regex in copy command
shivaram
committed
[SPARK-18745][SQL] Fix signed integer overflow due to toInt cast

kiszk
authored and
hvanhovell
committed

Commits on Dec 10, 2016

[SPARK-18812][MLLIB] explain "Spark ML"
mengxr
committed
[SPARK-18807][SPARKR] Should suppress output print for calls to JVM methods with void return values

felixcheung
authored and
shivaram
committed
[SPARK-18811] StreamSource resolution should happen in stream execution thread

brkyvz
authored and
zsxwing
committed
[SPARK-17460][SQL] Make sure sizeInBytes in Statistics will not overflow

huaxingao
authored and
cloud-fan
committed
[MINOR][DOCS] Remove Apache Spark Wiki address

dongjoon-hyun
authored and
srowen
committed
[SPARK-3359][DOCS] Fix greater-than symbols in Javadoc to allow building with Java 8

michalsenkyr
authored and
srowen
committed

Commits on Dec 11, 2016

[SPARK-18815][SQL] Fix NPE when collecting column stats for string/binary column having only null values

wangzhenhua
authored and
rxin
committed
[SQL][MINOR] simplify a test to fix the maven tests

cloud-fan
authored and
srowen
committed
[SPARK-18628][ML] Update Scala param and Python param to have quotes

krishnakalyan3
authored and
srowen
committed

Commits on Dec 12, 2016

[SPARK-18790][SS] Keep a general offset history of stream batches

Tyson Condie
authored and
zsxwing
committed
[DOCS][MINOR] Clarify Where AccumulatorV2s are Displayed

Bill Chambers
authored and
srowen
committed
[SPARK-18681][SQL] Fix filtering to compatible with partition keys of type int

wangyum
authored and
hvanhovell
committed
[SPARK-18810][SPARKR] SparkR install.spark does not work for RCs, snapshots

felixcheung
authored and
shivaram
committed

Commits on Dec 13, 2016

[SPARK-18796][SS] StreamingQueryManager should not block when starting a query

zsxwing
authored and
tdas
committed
[SPARK-18797][SPARKR] Update spark.logit in sparkr-vignettes

wangmiao1981
authored and
mengxr
committed
[SPARK-18835][SQL] Don't expose Guava types in the JavaTypeInference API.
Marcelo Vanzin
committed
[SPARK-18840][YARN] Avoid throw exception when getting token renewal interval in non HDFS security environment

jerryshao
authored and
Marcelo Vanzin
committed
[SPARK-18816][WEB UI] Executors Logs column only ran visibility check on initial table load

ajbozarth
authored and
srowen
committed
[SPARK-18843][CORE] Fix timeout in awaitResultInForkJoinSafely (branch 2.1, 2.0)
zsxwing
committed
[SPARK-18834][SS] Expose event time stats through StreamingQueryProgress
tdas
committed

Commits on Dec 14, 2016

Commits on Dec 15, 2016

Commits on Dec 16, 2016

Commits on Dec 17, 2016

[SPARK-18849][ML][SPARKR][DOC] vignettes final check reorg

felixcheung
authored and
Felix Cheung
committed

Commits on Dec 18, 2016

[SPARK-18918][DOC] Missing </td> in Configuration page

gatorsmile
authored and
srowen
committed
[SPARK-18827][CORE] Fix cannot read broadcast on disk

wangyum
authored and
srowen
committed

Commits on Dec 19, 2016

Commits on Dec 21, 2016

Commits on Dec 22, 2016

Commits on Dec 23, 2016

Commits on Dec 24, 2016

[SPARK-18837][WEBUI] Very long stage descriptions do not wrap in the UI

sarutak
authored and
srowen
committed

Commits on Dec 28, 2016

Commits on Dec 29, 2016

[SPARK-19003][DOCS] Add Java example in Spark Streaming Guide, section Design Patterns for using foreachRDD

adesharatushar
authored and
srowen
committed

Commits on Dec 30, 2016

[SPARK-19016][SQL][DOC] Document scalable partition handling
liancheng
committed

Commits on Jan 1, 2017

[SPARK-19050][SS][TESTS] Fix EventTimeWatermarkSuite 'delay in months and years handled correctly'
zsxwing
committed

Commits on Jan 2, 2017

[MINOR][DOC] Minor doc change for YARN credential providers

viirya
authored and
srowen
committed
[SPARK-18379][SQL] Make the parallelism of parallelPartitionDiscovery configurable.

genmao.ygm
authored and
srowen
committed
[SPARK-19041][SS] Fix code snippet compilation issues in Structured Streaming Programming Guide

lw-lin
authored and
zsxwing
committed

Commits on Jan 3, 2017

Commits on Jan 4, 2017

[SPARK-18877][SQL][BACKPORT-2.1] CSVInferSchema.inferField` on DecimalType should find a common type with `typeSoFar`

dongjoon-hyun
authored and
gatorsmile
committed

Commits on Jan 6, 2017

[SPARK-19033][CORE] Add admin acls for history server

jerryshao
authored and
Tom Graves
committed
[SPARK-19083] sbin/start-history-server.sh script use of $@ without quotes

zuotingbing
authored and
Marcelo Vanzin
committed
[SPARK-19074][SS][DOCS] Updated Structured Streaming Programming Guide for update mode and source/sink options
tdas
committed

Commits on Jan 7, 2017

Commits on Jan 8, 2017

[SPARK-18941][SQL][DOC] Add a new behavior document on `CREATE/DROP TABLE` with `LOCATION`

dongjoon-hyun
authored and
gatorsmile
committed

Commits on Jan 9, 2017

[SPARK-19127][DOCS] Update Rank Function Documentation

bllchmbrs
authored and
rxin
committed
[SPARK-19126][DOCS] Update Join Documentation Across Languages

bllchmbrs
authored and
Felix Cheung
committed
[SPARK-18903][SPARKR][BACKPORT-2.1] Add API to get SparkUI URL

felixcheung
authored and
Felix Cheung
committed
[SPARK-18952][BACKPORT] Regex strings not properly escaped in codegen for aggregations

brkyvz
authored and
JoshRosen
committed

Commits on Jan 10, 2017

[SPARK-16845][SQL] `GeneratedClass$SpecificOrdering` grows beyond 64 KB

lw-lin
authored and
cloud-fan
committed
[SPARK-19137][SQL] Fix `withSQLConf` to reset `OptionalConfigEntry` correctly

dongjoon-hyun
authored and
zsxwing
committed
[SPARK-19113][SS][TESTS] Set UncaughtExceptionHandler in onQueryStarted to ensure catching fatal errors during query initialization
zsxwing
committed
[SPARK-18997][CORE] Recommended upgrade libthrift to 0.9.3

srowen
authored and
Marcelo Vanzin
committed

Commits on Jan 11, 2017

[SPARK-19140][SS] Allow update mode for non-aggregation streaming queries
zsxwing
committed
[SPARK-19133][SPARKR][ML][BACKPORT-2.1] fix glm for Gamma, clarify glm family supported

felixcheung
authored and
Felix Cheung
committed
[SPARK-19130][SPARKR] Support setting literal value as column implicitly

felixcheung
authored and
shivaram
committed

Commits on Jan 12, 2017

Commits on Jan 13, 2017

Fix missing close-parens for In filter's toString

ash211
authored and
rxin
committed
[SPARK-18687][PYSPARK][SQL] Backward compatibility - creating a Dataframe on a new SQLContext object fails with a Derby error

vijoshi
authored and
cloud-fan
committed
[SPARK-19178][SQL] convert string of large numbers to int should return null
cloud-fan
committed
[SPARK-18335][SPARKR] createDataFrame to support numPartitions parameter

felixcheung
authored and
shivaram
committed
[SPARK-19180] [SQL] the offset of short should be 2 in OffHeapColumn

Yucai Yu
authored and
davies
committed

Commits on Jan 15, 2017

[SPARK-19120] Refresh Metadata Cache After Loading Hive Tables

gatorsmile
authored and
cloud-fan
committed

Commits on Jan 16, 2017

[SPARK-19092][SQL][BACKPORT-2.1] Save() API of DataFrameWriter should not scan all the saved files #16481

gatorsmile
authored and
cloud-fan
committed
[SPARK-19082][SQL] Make ignoreCorruptFiles work for Parquet

viirya
authored and
cloud-fan
committed
[SPARK-19232][SPARKR] Update Spark distribution download cache location on Windows

felixcheung
authored and
shivaram
committed

Commits on Jan 18, 2017

[SPARK-19066][SPARKR][BACKPORT-2.1] LDA doesn't set optimizer correctly

wangmiao1981
authored and
Felix Cheung
committed
[SPARK-19231][SPARKR] add error handling for download and untar for Spark release

felixcheung
authored and
Felix Cheung
committed
[SPARK-19113][SS][TESTS] Ignore StreamingQueryException thrown from awaitInitialization to avoid breaking tests
zsxwing
committed
[SPARK-19168][STRUCTURED STREAMING] StateStore should be aborted upon error

lw-lin
authored and
zsxwing
committed

Commits on Jan 20, 2017

Commits on Jan 21, 2017

[SPARK-18589][SQL] Fix Python UDF accessing attributes from both side of join

Davies Liu
authored and
hvanhovell
committed
[SPARK-19267][SS] Fix a race condition when stopping StateStore

zsxwing
authored and
tdas
committed

Commits on Jan 22, 2017

[SPARK-19155][ML] MLlib GeneralizedLinearRegression family and link should case insensitive
yanboliang
committed

Commits on Jan 23, 2017

[SPARK-19155][ML] Make family case insensitive in GLM

actuaryzhang
authored and
yanboliang
committed
[SPARK-19306][CORE] Fix inconsistent state in DiskBlockObject when expection occurred

jerryshao
authored and
Marcelo Vanzin
committed

Commits on Jan 24, 2017

[SPARK-9435][SQL] Reuse function in Java UDF to correctly support expressions that require equality comparison between ScalaUDF

HyukjinKwon
authored and
gatorsmile
committed
[SPARK-19268][SS] Disallow adaptive query execution for streaming queries
zsxwing
committed
[SPARK-18823][SPARKR] add support for assigning to column

felixcheung
authored and
Felix Cheung
committed
[SPARK-16473][MLLIB] Fix BisectingKMeans Algorithm failing in edge case

imatiach-msft
authored and
srowen
committed
[SPARK-19017][SQL] NOT IN subquery with more than one column may return incorrect results

nsyca
authored and
hvanhovell
committed

Commits on Jan 25, 2017

[SPARK-19330][DSTREAMS] Also show tooltip for successful batches

lw-lin
authored and
zsxwing
committed
[SPARK-16046][DOCS] Aggregations in the Spark SQL programming guide

aokolnychyi
authored and
gatorsmile
committed
[SPARK-18750][YARN] Avoid using "mapValues" when allocating containers.

Marcelo Vanzin
authored and
Tom Graves
committed
[SPARK-18863][SQL] Output non-aggregate expressions without GROUP BY in a subquery does not yield an error

nsyca
authored and
hvanhovell
committed
[SPARK-19307][PYSPARK] Make sure user conf is propagated to SparkContext.
Marcelo Vanzin
committed
[SPARK-18750][YARN] Follow up: move test to correct directory in 2.1 branch.
Marcelo Vanzin
committed
[SPARK-19064][PYSPARK] Fix pip installing of sub components
holdenk
committed

Commits on Jan 26, 2017

Commits on Jan 27, 2017

[SPARK-19220][UI] Make redirection to HTTPS apply to all URIs. (branch-2.1)
Marcelo Vanzin
committed
[SPARK-18788][SPARKR] Add API for getNumPartitions

felixcheung
authored and
Felix Cheung
committed
[SPARK-19333][SPARKR] Add Apache License headers to R files

felixcheung
authored and
Felix Cheung
committed
[SPARK-19324][SPARKR] Spark VJM stdout output is getting dropped in SparkR

felixcheung
authored and
shivaram
committed

Commits on Jan 31, 2017

doc fix
felixcheung
committed