[SPARK-2873] [SQL] using ExternalAppendOnlyMap to resolve OOM when aggregating#1822

Closed

guowei2 wants to merge 157 commits intoapache:masterfrom

guowei2:sql-memory-patch

Commits on Jul 3, 2014

SITUATION: ShuffledDStream run tasks whether dstream has partition items or not
guowei
committed

Commits on Jul 10, 2014

Commits on Jul 11, 2014

DStream run tasks only when dstream has partition items
guowei
committed

Commits on Aug 6, 2014

Commits on Aug 7, 2014

[SPARK-2887] fix bug of countApproxDistinct() when have more than one partition

davies
authored and
pwendell
committed
[SPARK-2851] [mllib] DecisionTree Python consistency update

jkbradley
authored and
mengxr
committed
SPARK-2879 part 2 [BUILD] Use HTTPS to access Maven Central and other repos

srowen
authored and
pwendell
committed
[mllib] DecisionTree Strategy parameter checks

jkbradley
authored and
mengxr
committed
[SPARK-2852][MLLIB] Separate model from IDF/StandardScaler algorithms
mengxr
committed
SPARK-2905 Fixed path sbin => bin

dosoft
authored and
pwendell
committed
SPARK-2899 Doc generation is back to working in new SBT Build.

ScrapCodes
authored and
pwendell
committed

Commits on Aug 8, 2014

SPARK-2787: Make sort-based shuffle write files directly when there's no sorting/aggregation and # partitions is small

mateiz
authored and
rxin
committed
SPARK-2565. Update ShuffleReadMetrics as blocks are fetched

sryza
authored and
pwendell
committed
[SPARK-2904] Remove non-used local variable in SparkSubmitArguments

sarutak
authored and
pwendell
committed
[SPARK-2911]: provide rdd.parent[T](j) to obtain jth parent RDD

erikerlandson
authored and
rxin
committed
[SPARK-2888] [SQL] Fix addColumnMetadataToConf in HiveTableScan

yhuai
authored and
marmbrus
committed
[SPARK-2908] [SQL] JsonRDD.nullTypeToStringType does not convert all NullType to StringType

yhuai
authored and
marmbrus
committed
[SPARK-2877] [SQL] MetastoreRelation should use SparkClassLoader when creating the tableDesc

yhuai
authored and
marmbrus
committed
[SPARK-2919] [SQL] Basic support for analyze command in HiveQl

yhuai
authored and
marmbrus
committed
[SPARK-2700] [SQL] Hidden files (such as .impala_insert_staging) should be filtered out by sqlContext.parquetFile

chutium
authored and
marmbrus
committed
[SPARK-1997][MLLIB] update breeze to 0.9
mengxr
committed
[SPARK-2897][SPARK-2920]TorrentBroadcast does use the serializer class specified in the spark option "spark.serializer"

witgo
authored and
rxin
committed

Commits on Aug 9, 2014

[Web UI]Make decision order of Worker's WebUI port consistent with Master's

WangTaoTheTonic
authored and
rxin
committed
[SPARK-2911] apply parent[T](j) to clarify UnionRDD code

erikerlandson
authored and
rxin
committed
[SPARK-2635] Fix race condition at SchedulerBackend.isReady in standalone mode

li-zhihui
authored and
pwendell
committed
[SPARK-2861] Fix Doc comment of histogram method

Chandan Kumar
authored and
pwendell
committed

Commits on Aug 10, 2014

[SPARK-1766] sorted functions to meet pedantic requirements

Chris Cope
authored and
pwendell
committed
[SPARK-2894] spark-shell doesn't accept flags

sarutak
authored and
pwendell
committed
Updated Spark SQL README to include the hive-thriftserver module
rxin
committed
Turn UpdateBlockInfo into case class.
rxin
committed
Remove extra semicolon in Task.scala

witgo
authored and
rxin
committed
[SPARK-2950] Add gc time and shuffle write time to JobLogger
shivaram
committed
[SPARK-2898] [PySpark] fix bugs in deamon.py

davies
authored and
JoshRosen
committed
[SPARK-2937] Separate out samplyByKeyExact as its own API in PairRDDFunction

dorx
authored and
mengxr
committed

Commits on Aug 11, 2014

Commits on Aug 12, 2014

[SPARK-2931] In TaskSetManager, reset currentLocalityIndex after recomputing locality levels
JoshRosen
committed
[SPARK-2515][mllib] Chi Squared test

dorx
authored and
mengxr
committed
[SPARK-2934][MLlib] Adding LogisticRegressionWithLBFGS Interface

DB Tsai
authored and
mengxr
committed
[SPARK-2844][SQL] Correctly set JVM HiveContext if it is passed into Python HiveContext constructor

ahirreddy
authored and
marmbrus
committed
[SPARK-2590][SQL] Added option to handle incremental collection, disabled by default

liancheng
authored and
marmbrus
committed
[sql]use SparkSQLEnv.stop() in ShutdownHook

scwf
authored and
marmbrus
committed
[SQL] A tiny refactoring in HiveContext#analyze

yhuai
authored and
marmbrus
committed
[SPARK-2965][SQL] Fix HashOuterJoin output nullabilities.

ueshin
authored and
marmbrus
committed
[SPARK-2968][SQL] Fix nullabilities of Explode.

ueshin
authored and
marmbrus
committed
[SPARK-2650][SQL] Build column buffers in smaller batches
marmbrus
committed
[SQL] [SPARK-2826] Reduce the memory copy while building the hashmap for HashOuterJoin

chenghao-intel
authored and
marmbrus
committed
[SPARK-2923][MLLIB] Implement some basic BLAS routines
mengxr
committed
[MLlib] Correctly set vectorSize and alpha

Ishiihara
authored and
mengxr
committed
fix flaky tests

davies
authored and
marmbrus
committed

Commits on Aug 13, 2014

SPARK-2830 [MLlib]: re-organize mllib documentation

atalwalkar
authored and
mengxr
committed
Choose OnHeapAggregation or ExternalAggregation base on spark.sql.aggregate.external in SQLConf
guowei2
committed
Choose OnHeapAggregation or ExternalAggregation base on spark.sql.aggregate.external in SQLConf
guowei2
committed
[SPARK-2953] Allow using short names for io compression codecs
rxin
committed
Use transferTo when copy merge files in ExternalSorter

colorant
authored and
rxin
committed
[SPARK-1777 (partial)] bugfix: make size of requested memory correctly

liyezhang556520
authored and
rxin
committed
[SPARK-2993] [MLLib] colStats (wrapper around MultivariateStatisticalSummary) in Statistics

dorx
authored and
mengxr
committed
[SPARK-2963] [SQL] There no documentation about building to use HiveServer and CLI for SparkSQL

sarutak
authored and
marmbrus
committed
[SPARK-3013] [SQL] [PySpark] convert array into list

davies
authored and
marmbrus
committed
[SPARK-2983] [PySpark] improve performance of sortByKey()

davies
authored and
mateiz
committed
[MLLIB] use Iterator.fill instead of Array.fill
mengxr
committed
[SPARK-3004][SQL] Added null checking when retrieving row set

liancheng
authored and
marmbrus
committed
[SPARK-2817] [SQL] add "show create table" support

tianyi
authored and
marmbrus
committed

Commits on Aug 14, 2014

[SPARK-2994][SQL] Support for udfs that take complex types
marmbrus
committed
[SPARK-2650][SQL] More precise initial buffer size estimation for in-memory column buffer

liancheng
authored and
marmbrus
committed
[SPARK-2935][SQL]Fix parquet predicate push down bug
marmbrus
committed
[SPARK-2970] [SQL] spark-sql script ends with IOException when EventLogging is enabled

sarutak
authored and
marmbrus
committed
[SPARK-2986] [SQL] fixed: setting properties does not effect

guowei
authored and
marmbrus
committed
SPARK-3020: Print completed indices rather than tasks in web UI

pwendell
authored and
rxin
committed
[SPARK-3006] Failed to execute spark-shell in Windows OS

tsudukim
authored and
andrewor14
committed
[Docs] Add missing <code> tags (minor)

andrewor14
authored and
rxin
committed
[SPARK-2995][MLLIB] add ALS.setIntermediateRDDStorageLevel
mengxr
committed
[SPARK-3029] Disable local execution of Spark jobs by default

aarondav
authored and
rxin
committed
SPARK-2893: Do not swallow Exceptions when running a custom kryo registrator

GrahamDennis
authored and
rxin
committed
[SPARK-3011][SQL] _temporary directory should be filtered out by sqlContext.parquetFile

josephsu
authored and
marmbrus
committed
[SPARK-2927][SQL] Add a conf to configure if we always read Binary columns stored in Parquet as String columns

yhuai
authored and
marmbrus
committed
[SQL] Python JsonRDD UTF8 Encoding Fix

ahirreddy
authored and
marmbrus
committed
[SPARK-2925] [sql]fix spark-sql and start-thriftserver shell bugs when set --driver-java-options

scwf
authored and
marmbrus
committed
Minor cleanup of metrics.Source
rxin
committed
[SPARK-2979][MLlib] Improve the convergence rate by minimizing the condition number

DB Tsai
authored and
mengxr
committed
Revert [SPARK-3011][SQL] _temporary directory should be filtered out by sqlContext.parquetFile
marmbrus
committed
SPARK-3009: Reverted readObject method in ApplicationInfo so that Applic...

jacek-lewandowski
authored and
andrewor14
committed
Make dev/mima runnable on Mac OS X.
rxin
committed

Commits on Aug 15, 2014

[SPARK-3027] TaskContext: tighten visibility and provide Java friendly callback API
rxin
committed
[SPARK-2468] Netty based block server / client module
rxin
committed
[SPARK-2736] PySpark converter and example script for reading Avro files

kanzhang
authored and
mateiz
committed
[SPARK-2912] [Spark QA] Include commit hash in Spark QA messages

nchammas
authored and
pwendell
committed
SPARK-2955 [BUILD] Test code fails to compile with "mvn compile" without "install"

srowen
authored and
pwendell
committed
Add caching information to rdd.toDebugString

Nathan Kronenfeld
authored and
pwendell
committed
[SPARK-2924] remove default args to overloaded methods

avati
authored and
pwendell
committed
Revert "[SPARK-2468] Netty based block server / client module"
pwendell
committed
SPARK-3028. sparkEventToJson should support SparkListenerExecutorMetrics...

sryza
authored and
pwendell
committed
[SPARK-3022] [SPARK-3041] [mllib] Call findBins once per level + unordered feature bug fix

jkbradley
authored and
mengxr
committed

Commits on Aug 16, 2014

Commits on Aug 17, 2014

In the stop method of ConnectionManager to cancel the ackTimeoutMonitor

witgo
authored and
JoshRosen
committed
[SPARK-3077][MLLIB] fix some chisq-test
mengxr
committed
[SPARK-3042] [mllib] DecisionTree Filter top-down instead of bottom-up

jkbradley
authored and
mengxr
committed

Commits on Aug 18, 2014

Commits on Aug 19, 2014

fix Big-ass commit
guowei
committed