Update#3

Merged

YanTangZhai merged 163 commits intoYanTangZhai:masterfrom

Aug 20, 2014

Commits on Aug 6, 2014

[SPARK-2875] [PySpark] [SQL] handle null in schemaRDD()

davies
authored and
marmbrus
committed
[SPARK-2678][Core][SQL] A workaround for SPARK-2678

liancheng
authored and
pwendell
committed
[SPARK-2627] [PySpark] have the build enforce PEP 8 automatically

nchammas
authored and
rxin
committed
SPARK-2566. Update ShuffleWriteMetrics incrementally

sryza
authored and
pwendell
committed
[SPARK-2852][MLLIB] API consistency for `mllib.feature`
mengxr
committed
[PySpark] Add blanklines to Python docstrings so example code renders correctly

rnowling
authored and
mengxr
committed
[HOTFIX][Streaming] Handle port collisions in flume polling test

andrewor14
authored and
tdas
committed
SPARK-2882: Spark build now checks local maven cache for dependencies

GregOwen
authored and
pwendell
committed

Commits on Aug 7, 2014

[SPARK-2583] ConnectionManager error reporting

sarutak
authored and
pwendell
committed
SPARK-2879 [BUILD] Use HTTPS to access Maven Central and other repos

srowen
authored and
pwendell
committed
HOTFIX: Support custom Java 7 location
pwendell
committed
[SPARK-2887] fix bug of countApproxDistinct() when have more than one partition

davies
authored and
pwendell
committed
[SPARK-2851] [mllib] DecisionTree Python consistency update

jkbradley
authored and
mengxr
committed
SPARK-2879 part 2 [BUILD] Use HTTPS to access Maven Central and other repos

srowen
authored and
pwendell
committed
[mllib] DecisionTree Strategy parameter checks

jkbradley
authored and
mengxr
committed
[SPARK-2852][MLLIB] Separate model from IDF/StandardScaler algorithms
mengxr
committed
SPARK-2905 Fixed path sbin => bin

dosoft
authored and
pwendell
committed
SPARK-2899 Doc generation is back to working in new SBT Build.

ScrapCodes
authored and
pwendell
committed

Commits on Aug 8, 2014

SPARK-2787: Make sort-based shuffle write files directly when there's no sorting/aggregation and # partitions is small

mateiz
authored and
rxin
committed
SPARK-2565. Update ShuffleReadMetrics as blocks are fetched

sryza
authored and
pwendell
committed
[SPARK-2904] Remove non-used local variable in SparkSubmitArguments

sarutak
authored and
pwendell
committed
[SPARK-2911]: provide rdd.parent[T](j) to obtain jth parent RDD

erikerlandson
authored and
rxin
committed
[SPARK-2888] [SQL] Fix addColumnMetadataToConf in HiveTableScan

yhuai
authored and
marmbrus
committed
[SPARK-2908] [SQL] JsonRDD.nullTypeToStringType does not convert all NullType to StringType

yhuai
authored and
marmbrus
committed
[SPARK-2877] [SQL] MetastoreRelation should use SparkClassLoader when creating the tableDesc

yhuai
authored and
marmbrus
committed
[SPARK-2919] [SQL] Basic support for analyze command in HiveQl

yhuai
authored and
marmbrus
committed
[SPARK-2700] [SQL] Hidden files (such as .impala_insert_staging) should be filtered out by sqlContext.parquetFile

chutium
authored and
marmbrus
committed
[SPARK-1997][MLLIB] update breeze to 0.9
mengxr
committed
[SPARK-2897][SPARK-2920]TorrentBroadcast does use the serializer class specified in the spark option "spark.serializer"

witgo
authored and
rxin
committed

Commits on Aug 9, 2014

[Web UI]Make decision order of Worker's WebUI port consistent with Master's

WangTaoTheTonic
authored and
rxin
committed
[SPARK-2911] apply parent[T](j) to clarify UnionRDD code

erikerlandson
authored and
rxin
committed
[SPARK-2635] Fix race condition at SchedulerBackend.isReady in standalone mode

li-zhihui
authored and
pwendell
committed
[SPARK-2861] Fix Doc comment of histogram method

Chandan Kumar
authored and
pwendell
committed

Commits on Aug 10, 2014

[SPARK-1766] sorted functions to meet pedantic requirements

Chris Cope
authored and
pwendell
committed
[SPARK-2894] spark-shell doesn't accept flags

sarutak
authored and
pwendell
committed
Updated Spark SQL README to include the hive-thriftserver module
rxin
committed
Turn UpdateBlockInfo into case class.
rxin
committed
Remove extra semicolon in Task.scala

witgo
authored and
rxin
committed
[SPARK-2950] Add gc time and shuffle write time to JobLogger
shivaram
committed
[SPARK-2898] [PySpark] fix bugs in deamon.py

davies
authored and
JoshRosen
committed
[SPARK-2937] Separate out samplyByKeyExact as its own API in PairRDDFunction

dorx
authored and
mengxr
committed

Commits on Aug 11, 2014

Commits on Aug 12, 2014

[SPARK-2931] In TaskSetManager, reset currentLocalityIndex after recomputing locality levels
JoshRosen
committed
[SPARK-2515][mllib] Chi Squared test

dorx
authored and
mengxr
committed
[SPARK-2934][MLlib] Adding LogisticRegressionWithLBFGS Interface

DB Tsai
authored and
mengxr
committed
[SPARK-2844][SQL] Correctly set JVM HiveContext if it is passed into Python HiveContext constructor

ahirreddy
authored and
marmbrus
committed
[SPARK-2590][SQL] Added option to handle incremental collection, disabled by default

liancheng
authored and
marmbrus
committed
[sql]use SparkSQLEnv.stop() in ShutdownHook

scwf
authored and
marmbrus
committed
[SQL] A tiny refactoring in HiveContext#analyze

yhuai
authored and
marmbrus
committed
[SPARK-2965][SQL] Fix HashOuterJoin output nullabilities.

ueshin
authored and
marmbrus
committed
[SPARK-2968][SQL] Fix nullabilities of Explode.

ueshin
authored and
marmbrus
committed
[SPARK-2650][SQL] Build column buffers in smaller batches
marmbrus
committed
[SQL] [SPARK-2826] Reduce the memory copy while building the hashmap for HashOuterJoin

chenghao-intel
authored and
marmbrus
committed
[SPARK-2923][MLLIB] Implement some basic BLAS routines
mengxr
committed
[MLlib] Correctly set vectorSize and alpha

Ishiihara
authored and
mengxr
committed
fix flaky tests

davies
authored and
marmbrus
committed

Commits on Aug 13, 2014

SPARK-2830 [MLlib]: re-organize mllib documentation

atalwalkar
authored and
mengxr
committed
[SPARK-2953] Allow using short names for io compression codecs
rxin
committed
Use transferTo when copy merge files in ExternalSorter

colorant
authored and
rxin
committed
[SPARK-1777 (partial)] bugfix: make size of requested memory correctly

liyezhang556520
authored and
rxin
committed
[SPARK-2993] [MLLib] colStats (wrapper around MultivariateStatisticalSummary) in Statistics

dorx
authored and
mengxr
committed
[SPARK-2963] [SQL] There no documentation about building to use HiveServer and CLI for SparkSQL

sarutak
authored and
marmbrus
committed
[SPARK-3013] [SQL] [PySpark] convert array into list

davies
authored and
marmbrus
committed
[SPARK-2983] [PySpark] improve performance of sortByKey()

davies
authored and
mateiz
committed
[MLLIB] use Iterator.fill instead of Array.fill
mengxr
committed
[SPARK-3004][SQL] Added null checking when retrieving row set

liancheng
authored and
marmbrus
committed
[SPARK-2817] [SQL] add "show create table" support

tianyi
authored and
marmbrus
committed

Commits on Aug 14, 2014

[SPARK-2994][SQL] Support for udfs that take complex types
marmbrus
committed
[SPARK-2650][SQL] More precise initial buffer size estimation for in-memory column buffer

liancheng
authored and
marmbrus
committed
[SPARK-2935][SQL]Fix parquet predicate push down bug
marmbrus
committed
[SPARK-2970] [SQL] spark-sql script ends with IOException when EventLogging is enabled

sarutak
authored and
marmbrus
committed
[SPARK-2986] [SQL] fixed: setting properties does not effect

guowei
authored and
marmbrus
committed
SPARK-3020: Print completed indices rather than tasks in web UI

pwendell
authored and
rxin
committed
[SPARK-3006] Failed to execute spark-shell in Windows OS

tsudukim
authored and
andrewor14
committed
[Docs] Add missing <code> tags (minor)

andrewor14
authored and
rxin
committed
[SPARK-2995][MLLIB] add ALS.setIntermediateRDDStorageLevel
mengxr
committed
[SPARK-3029] Disable local execution of Spark jobs by default

aarondav
authored and
rxin
committed
SPARK-2893: Do not swallow Exceptions when running a custom kryo registrator

GrahamDennis
authored and
rxin
committed
[SPARK-3011][SQL] _temporary directory should be filtered out by sqlContext.parquetFile

josephsu
authored and
marmbrus
committed
[SPARK-2927][SQL] Add a conf to configure if we always read Binary columns stored in Parquet as String columns

yhuai
authored and
marmbrus
committed
[SQL] Python JsonRDD UTF8 Encoding Fix

ahirreddy
authored and
marmbrus
committed
[SPARK-2925] [sql]fix spark-sql and start-thriftserver shell bugs when set --driver-java-options

scwf
authored and
marmbrus
committed
Minor cleanup of metrics.Source
rxin
committed
[SPARK-2979][MLlib] Improve the convergence rate by minimizing the condition number

DB Tsai
authored and
mengxr
committed
Revert [SPARK-3011][SQL] _temporary directory should be filtered out by sqlContext.parquetFile
marmbrus
committed
SPARK-3009: Reverted readObject method in ApplicationInfo so that Applic...

jacek-lewandowski
authored and
andrewor14
committed
Make dev/mima runnable on Mac OS X.
rxin
committed

Commits on Aug 15, 2014

[SPARK-3027] TaskContext: tighten visibility and provide Java friendly callback API
rxin
committed
[SPARK-2468] Netty based block server / client module
rxin
committed
[SPARK-2736] PySpark converter and example script for reading Avro files

kanzhang
authored and
mateiz
committed
[SPARK-2912] [Spark QA] Include commit hash in Spark QA messages

nchammas
authored and
pwendell
committed
SPARK-2955 [BUILD] Test code fails to compile with "mvn compile" without "install"

srowen
authored and
pwendell
committed
Add caching information to rdd.toDebugString

Nathan Kronenfeld
authored and
pwendell
committed
[SPARK-2924] remove default args to overloaded methods

avati
authored and
pwendell
committed
Revert "[SPARK-2468] Netty based block server / client module"
pwendell
committed
SPARK-3028. sparkEventToJson should support SparkListenerExecutorMetrics...

sryza
authored and
pwendell
committed
[SPARK-3022] [SPARK-3041] [mllib] Call findBins once per level + unordered feature bug fix

jkbradley
authored and
mengxr
committed

Commits on Aug 16, 2014

Commits on Aug 17, 2014

In the stop method of ConnectionManager to cancel the ackTimeoutMonitor

witgo
authored and
JoshRosen
committed
[SPARK-3077][MLLIB] fix some chisq-test
mengxr
committed
[SPARK-3042] [mllib] DecisionTree Filter top-down instead of bottom-up

jkbradley
authored and
mengxr
committed

Commits on Aug 18, 2014

SPARK-2881. Upgrade snappy-java to 1.1.1.3.
pwendell
committed
Revert "[SPARK-2970] [SQL] spark-sql script ends with IOException when EventLogging is enabled"
marmbrus
committed
[SQL] Improve debug logging and toStrings.
marmbrus
committed
[SPARK-1981] updated streaming-kinesis.md

cfregly
authored and
tdas
committed
[HOTFIX][STREAMING] Allow the JVM/Netty to decide which port to bind to in Flume Polling Tests.

harishreedharan
authored and
tdas
committed
[SPARK-3087][MLLIB] fix col indexing bug in chi-square and add a check for number of distinct values
mengxr
committed
SPARK-2884: Create binary builds in parallel with release script.
pwendell
committed
SPARK-2900. aggregate inputBytes per stage

sryza
authored and
pwendell
committed
[SPARK-3097][MLlib] Word2Vec performance improvement

Ishiihara
authored and
mengxr
committed
[SPARK-2842][MLlib]Word2Vec documentation

Ishiihara
authored and
mengxr
committed
[MLlib] Remove transform(dataset: RDD[String]) from Word2Vec public API

Ishiihara
authored and
mengxr
committed
SPARK-3093 : masterLock in Worker is no longer need

CrazyJvm
authored and
aarondav
committed
[SPARK-2862] histogram method fails on some choices of bucketCount

Chandan Kumar
authored and
mengxr
committed
SPARK-3096: Include parquet hive serde by default in build

pwendell
authored and
marmbrus
committed
[SPARK-3084] [SQL] Collect broadcasted tables in parallel in joins

mateiz
authored and
marmbrus
committed
[SPARK-3085] [SQL] Use compact data structures in SQL joins

mateiz
authored and
marmbrus
committed
SPARK-3025 [SQL]: Allow JDBC clients to set a fair scheduler pool

pwendell
authored and
marmbrus
committed
[SPARK-3091] [SQL] Add support for caching metadata on Parquet files

mateiz
authored and
marmbrus
committed
[SPARK-2406][SQL] Initial support for using ParquetTableScan to read HiveMetaStore tables.
marmbrus
committed
[SPARK-2169] Don't copy appName / basePath everywhere.

Marcelo Vanzin
authored and
andrewor14
committed
Removed .travis.yml file since we are not using Travis.
rxin
committed
[SPARK-3103] [PySpark] fix saveAsTextFile() with utf-8

davies
authored and
JoshRosen
committed
[SPARK-2718] [yarn] Handle quotes and other characters in user args.

Marcelo Vanzin
authored and
andrewor14
committed
[mllib] DecisionTree: treeAggregate + Python example bug fix

jkbradley
authored and
mengxr
committed

Commits on Aug 19, 2014

Commits on Aug 20, 2014

[SPARK-2468] Netty based block server / client module
rxin
committed
[MLLIB] minor update to word2vec
mengxr
committed
[SPARK-3112][MLLIB] Add documentation and example for StreamingLR

freeman-lab
authored and
mengxr
committed
[SQL] add note of use synchronizedMap in SQLConf

scwf
authored and
rxin
committed
[SPARK-3130][MLLIB] detect negative values in naive Bayes
mengxr
committed
[HOTFIX][Streaming][MLlib] use temp folder for checkpoint
mengxr
committed
[SPARK-3119] Re-implementation of TorrentBroadcast.
rxin
committed
[SPARK-3142][MLLIB] output shuffle data directly in Word2Vec
mengxr
committed
[SPARK-2974] [SPARK-2975] Fix two bugs related to spark.local.dirs

JoshRosen
authored and
pwendell
committed
[DOCS] Fixed wrong links

giwa
authored and
rxin
committed
[SPARK-3141] [PySpark] fix sortByKey() with take()

davies
authored and
pwendell
committed
[SPARK-3054][STREAMING] Add unit tests for Spark Sink.

harishreedharan
authored and
tdas
committed