[SPARK-17620][SQL] Determine Serde by hive.default.fileformat when Creating Hive Serde Tables #15190

dilipbiswal · 2016-09-21T23:01:09Z

What changes were proposed in this pull request?

Make sure the hive.default.fileformat is used to when creating the storage format metadata.

Output

scala> spark.sql("SET hive.default.fileformat=orc")
res1: org.apache.spark.sql.DataFrame = [key: string, value: string]

scala> spark.sql("CREATE TABLE tmp_default(id INT)")
res2: org.apache.spark.sql.DataFrame = []

Before

scala> spark.sql("DESC FORMATTED tmp_default").collect.foreach(println)
..
[# Storage Information,,]
[SerDe Library:,org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe,]
[InputFormat:,org.apache.hadoop.hive.ql.io.orc.OrcInputFormat,]
[OutputFormat:,org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat,]
[Compressed:,No,]
[Storage Desc Parameters:,,]
[  serialization.format,1,]

After

scala> spark.sql("DESC FORMATTED tmp_default").collect.foreach(println)
..
[# Storage Information,,]
[SerDe Library:,org.apache.hadoop.hive.ql.io.orc.OrcSerde,]
[InputFormat:,org.apache.hadoop.hive.ql.io.orc.OrcInputFormat,]
[OutputFormat:,org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat,]
[Compressed:,No,]
[Storage Desc Parameters:,,]
[  serialization.format,1,]

How was this patch tested?

Added new tests to HiveDDLCommandSuite

SparkQA · 2016-09-22T01:05:27Z

Test build #65742 has finished for PR 15190 at commit 042a94e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2016-09-22T01:11:08Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveDDLCommandSuite.scala

    assert(partition2.get.apply("c") == "1" && partition2.get.apply("d") == "2")
  }

+  test("Test default fileformat") {


Please update the test case name to Test the default fileformat for Hive-serde tables

gatorsmile · 2016-09-22T01:14:41Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveDDLCommandSuite.scala

+        s"""
+           |CREATE TABLE IF NOT EXISTS fileformat_test (id int)
+        """.stripMargin
+      val (desc, exists) = extractTableDesc(s1)


val (desc, exists) = extractTableDesc("CREATE TABLE IF NOT EXISTS fileformat_test (id int)")

gatorsmile · 2016-09-22T01:14:59Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveDDLCommandSuite.scala

+        s"""
+           |CREATE TABLE IF NOT EXISTS fileformat_test (id int)
+        """.stripMargin
+      val (desc, exists) = extractTableDesc(s1)


The same here.

@gatorsmile Thanks !! I have updated as per your comments.

gatorsmile · 2016-09-22T01:37:27Z

Please update the PR description. This is not for orc only

SparkQA · 2016-09-22T01:44:52Z

Test build #65744 has finished for PR 15190 at commit f60e760.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-09-22T03:16:16Z

Test build #65752 has finished for PR 15190 at commit f2b93de.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

viirya · 2016-09-22T05:07:46Z

sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala

          .orElse(Some("org.apache.hadoop.mapred.TextInputFormat")),
        outputFormat = defaultHiveSerde.flatMap(_.outputFormat)
          .orElse(Some("org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat")),
-        // Note: Keep this unspecified because we use the presence of the serde to decide


I think this is kept as unspecified because it is intended to write the table with Hive write path. If we specify serde here, it will be converted to datasource table. Is it ok? cc @cloud-fan

cc @yhuai to confirm

The comment is not valid now. This was removed by the PR: #13386 (See the code changes made in HiveMetastoreCatalog.scala)

The current checking conditions are based on ctx.createFileFormat and ctx.rowFormat. Thus, I think this PR looks ok. : )

@viirya @cloud-fan Actually i am not sure, if the above comment is in sync with the code. When we had this comment, we used to have CreateTableAsSelectLogicalPlan to represent the CTAS case and we used to check for serde's presence to determine whether or not to convert it to a data source table like following.

if (sessionState.convertCTAS && table.storage.serde.isEmpty) { // Do the conversion when spark.sql.hive.convertCTAS is true and the query // does not specify any storage format (file format and storage handler). if (table.identifier.database.isDefined) { throw new AnalysisException( "Cannot specify database name in a CTAS statement " + "when spark.sql.hive.convertCTAS is set to true.") } val mode = if (allowExisting) SaveMode.Ignore else SaveMode.ErrorIfExists CreateTableUsingAsSelect( TableIdentifier(desc.identifier.table), conf.defaultDataSourceName, temporary = false, Array.empty[String], bucketSpec = None, mode, options = Map.empty[String, String], child ) } else { val desc = if (table.storage.serde.isEmpty) { // add default serde table.withNewStorage( serde = Some("org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe")) } else { table }

I think this code has changed and moved to SparkSqlParser ?

yeah. looks ok now.

gatorsmile · 2016-09-22T05:48:17Z

PR title should be
Determine Serde by hive.default.fileformat when Creating Hive Serde Tables

viirya · 2016-09-22T07:17:17Z

sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala

-        // Note: Keep this unspecified because we use the presence of the serde to decide
-        // whether to convert a table created by CTAS to a datasource table.
-        serde = None,
+        serde = defaultHiveSerde.flatMap(_.serde),


In DataSinks strategy, we set a default serde for CreateTable if tableDesc.storage.serde.isEmpty. I think we should also remove it and add .orElse(Some("org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe")) here.

@viirya Please see my comment below

viirya · 2016-09-22T07:19:38Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveDDLCommandSuite.scala

    assert(partition2.get.apply("c") == "1" && partition2.get.apply("d") == "2")
  }

+  test("Test the default fileformat for Hive-serde tables") {


Let's add a test for withSQLConf("hive.default.fileformat" -> "")?

viirya · 2016-09-22T07:27:48Z

As we set default serde/inputFormat/outFormat now, I think we don't need to set default values for them in CreateHiveTableAsSelectCommand again. What do you think?

dilipbiswal · 2016-09-22T07:42:29Z

@viirya I think we can come here from multiple code paths like visitCreateTableUsing. I think we can come to DataSinks's CreateTable case without serde being set. Let me know what you think.

viirya · 2016-09-22T07:44:29Z

oh. right. Should we set default serde in visitCreateTableUsing?

viirya · 2016-09-22T07:47:35Z

Currently the default serde setting path looks scattered. I think it is better to avoid that.

dilipbiswal · 2016-09-22T07:49:51Z

@viirya In my understanding, thats the datasource table code path. I am not sure if we should look at hive.default.fileformat property to set the default storage for data source tables ? In my opinion, its probably better to leave it the way it is to make sure we always have a default setting in the common path so we don't miss it.

viirya · 2016-09-22T07:58:48Z

hmmm, as we prohibit CREATE TABLE USING" with hive serde table, can we reach DataSinks's CreateTable from the path if visitCreateTableUsing?

viirya · 2016-09-22T08:01:26Z

@dilipbiswal yeah. I see. I still think to set default serde at different places looks not good to me, especially in other places it is hard-coded.

viirya · 2016-09-22T08:02:17Z

@dilipbiswal nvm. It is not related to this change.

yhuai · 2016-09-22T15:06:17Z

Can you try CREATE TABLE tmp_default(id INT) as select .... and see if the table will be converted to parquet format?

dilipbiswal · 2016-09-22T18:31:19Z

@yhuai Hi Yin,

create table ... as select ... would respect the setting of hive.default.fileformat.

scala> spark.sql("SET hive.default.fileformat=parquet")
res14: org.apache.spark.sql.DataFrame = [key: string, value: string]

scala> spark.sql("CREATE TABLE tmp_default4 select * from tmp_default")
res11: org.apache.spark.sql.DataFrame = []

scala> spark.sql("DESC FORMATTED tmp_default4").collect.foreach(println)
...
[# Storage Information,,]
[SerDe Library:,org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe,]
[InputFormat:,org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat,]
[OutputFormat:,org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat,]
...

scala> spark.sql("SET hive.default.fileformat=orc")
res14: org.apache.spark.sql.DataFrame = [key: string, value: string]

scala> spark.sql("CREATE TABLE tmp_default5 select * from tmp_default")
res15: org.apache.spark.sql.DataFrame = []

scala> spark.sql("DESC FORMATTED tmp_default5").collect.foreach(println)
...
[# Storage Information,,]
[SerDe Library:,org.apache.hadoop.hive.ql.io.orc.OrcSerde,]
[InputFormat:,org.apache.hadoop.hive.ql.io.orc.OrcInputFormat,]
[OutputFormat:,org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat,]
...

gatorsmile · 2016-09-22T20:00:18Z

@dilipbiswal Are they converted to data source tables?

dilipbiswal · 2016-09-22T20:24:23Z

@gatorsmile Thanks.. didn't realize we wanted to find out the CTAS behaviour. Here is the result..
When convertCTAS is set to true, we create a data source table with parquet format.

scala> spark.sql("SET spark.sql.hive.convertCTAS=true")
res23: org.apache.spark.sql.DataFrame = [key: string, value: string]

scala> spark.sql("CREATE TABLE tmp_default6 as select * from tmp_default")
res24: org.apache.spark.sql.DataFrame = []

scala> spark.sql("DESC extended tmp_default6").collect.foreach(println)
[# Detailed Table Information,CatalogTable(
    Provider: parquet
    Storage(Location: file:/home/mygit/apache/spark/bin/spark-warehouse/tmp_default6, InputFormat: org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat, OutputFormat: org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat, Serde: org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe, Properties: [path=file:/user/hive/warehouse/tmp_default6, serialization.format=1])),]

gatorsmile · 2016-09-22T20:40:49Z

@dilipbiswal Based on your tests and the source codes, I think your fix does not break anything.

viirya · 2016-09-23T06:38:41Z

Looks good.

gatorsmile · 2016-10-11T00:10:36Z

cc @yhuai @cloud-fan Based on the above PR discussion, it sounds like this PR is ok to merge. What do you think? Thank you!

yhuai · 2016-10-11T16:47:59Z

If we have spark.sql.hive.convertCTAS=true and hive.default.fileformat=orc, what format will we use when we create a table through a CTAS statement?

dilipbiswal · 2016-10-11T18:09:13Z

@yhuai We will use Parquet format in your example. We look at SQL spark.sql.sources.default configuration to decide on the format to use ?

Here is the output for your perusal.

spark-sql> set spark.sql.hive.convertCTAS=true;
spark.sql.hive.convertCTAS  true
Time taken: 3.309 seconds, Fetched 1 row(s)
spark-sql> set hive.default.fileformat=orc;
hive.default.fileformat orc
Time taken: 0.053 seconds, Fetched 1 row(s)
spark-sql> CREATE TABLE IF NOT EXISTS test select 1 from foo;
spark-sql> describe formatted test;
...
# Storage Information       
SerDe Library:  org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe 
InputFormat:    org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat   
OutputFormat:   org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat

Now change spark.sql.sources.default=orc

spark-sql> set spark.sql.sources.default=orc;
spark.sql.sources.default   orc
spark-sql> CREATE TABLE IF NOT EXISTS test2 select 1 from foo;
Time taken: 0.451 seconds
spark-sql> describe formatted test2;
...
# Storage Information       
SerDe Library:  org.apache.hadoop.hive.ql.io.orc.OrcSerde   
InputFormat:    org.apache.hadoop.hive.ql.io.orc.OrcInputFormat 
OutputFormat:   org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat

Please let me know if you have any further questions.

gatorsmile · 2016-10-11T20:23:23Z

Added them to your test cases. : )

SparkQA · 2016-10-14T01:35:57Z

Test build #66919 has finished for PR 15190 at commit f32fe21.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2016-10-14T03:21:52Z

retest this please

SparkQA · 2016-10-14T05:05:23Z

Test build #66944 has finished for PR 15190 at commit f32fe21.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dilipbiswal · 2016-10-14T05:38:31Z

retest this please

SparkQA · 2016-10-14T07:50:41Z

Test build #66949 has finished for PR 15190 at commit f32fe21.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dilipbiswal · 2016-10-14T16:05:20Z

@gatorsmile @yhuai I have added a new test. Can we please take a look at this again ?

yhuai · 2016-10-14T17:39:21Z

@dilipbiswal That makes sense. Thank you for testing that. I do not have any other question.

gatorsmile · 2016-10-14T20:18:52Z

LGTM

gatorsmile · 2016-10-14T20:23:56Z

Merging to master! Thanks!

dilipbiswal · 2016-10-14T20:45:09Z

Thank you @yhuai @gatorsmile @cloud-fan @viirya @dafrista

yhuai · 2016-10-14T21:03:16Z

Seems this breaks the build. https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/job/spark-master-compile-sbt-scala-2.10/2843/console

yhuai · 2016-10-14T21:08:19Z

I am reverting this patch. Sorry.

yhuai · 2016-10-14T21:10:23Z

Reverted

dilipbiswal · 2016-10-14T21:13:07Z

@yhuai very sorry Yin.. Let me look what happened here.. Is there a way to open this pull request ? or i need to open a new one ?

gatorsmile · 2016-10-14T21:30:19Z

It sounds like the other merged PRs made a change and impact your code. Please resolve it. Thanks!

yhuai · 2016-10-14T21:31:58Z

Yea. Looks like so. No worries. Let's get it tested again.

dilipbiswal · 2016-10-14T22:51:41Z

@gatorsmile @yhuai Its due to difference between scala 2.10 and 2.11 compiler in the way they deal with named parameters. Looks like 2.10 is less forgiving :-) . I have opened #15495 with the fix.

…eating Hive Serde Tables ## What changes were proposed in this pull request? Reopens the closed PR #15190 (Please refer to the above link for review comments on the PR) Make sure the hive.default.fileformat is used to when creating the storage format metadata. Output ``` SQL scala> spark.sql("SET hive.default.fileformat=orc") res1: org.apache.spark.sql.DataFrame = [key: string, value: string] scala> spark.sql("CREATE TABLE tmp_default(id INT)") res2: org.apache.spark.sql.DataFrame = [] ``` Before ```SQL scala> spark.sql("DESC FORMATTED tmp_default").collect.foreach(println) .. [# Storage Information,,] [SerDe Library:,org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe,] [InputFormat:,org.apache.hadoop.hive.ql.io.orc.OrcInputFormat,] [OutputFormat:,org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat,] [Compressed:,No,] [Storage Desc Parameters:,,] [ serialization.format,1,] ``` After ```SQL scala> spark.sql("DESC FORMATTED tmp_default").collect.foreach(println) .. [# Storage Information,,] [SerDe Library:,org.apache.hadoop.hive.ql.io.orc.OrcSerde,] [InputFormat:,org.apache.hadoop.hive.ql.io.orc.OrcInputFormat,] [OutputFormat:,org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat,] [Compressed:,No,] [Storage Desc Parameters:,,] [ serialization.format,1,] ``` ## How was this patch tested? (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) Added new tests to HiveDDLCommandSuite, SQLQuerySuite Author: Dilip Biswal <[email protected]> Closes #15495 from dilipbiswal/orc2.

…eating Hive Serde Tables ## What changes were proposed in this pull request? Make sure the hive.default.fileformat is used to when creating the storage format metadata. Output ``` SQL scala> spark.sql("SET hive.default.fileformat=orc") res1: org.apache.spark.sql.DataFrame = [key: string, value: string] scala> spark.sql("CREATE TABLE tmp_default(id INT)") res2: org.apache.spark.sql.DataFrame = [] ``` Before ```SQL scala> spark.sql("DESC FORMATTED tmp_default").collect.foreach(println) .. [# Storage Information,,] [SerDe Library:,org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe,] [InputFormat:,org.apache.hadoop.hive.ql.io.orc.OrcInputFormat,] [OutputFormat:,org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat,] [Compressed:,No,] [Storage Desc Parameters:,,] [ serialization.format,1,] ``` After ```SQL scala> spark.sql("DESC FORMATTED tmp_default").collect.foreach(println) .. [# Storage Information,,] [SerDe Library:,org.apache.hadoop.hive.ql.io.orc.OrcSerde,] [InputFormat:,org.apache.hadoop.hive.ql.io.orc.OrcInputFormat,] [OutputFormat:,org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat,] [Compressed:,No,] [Storage Desc Parameters:,,] [ serialization.format,1,] ``` ## How was this patch tested? Added new tests to HiveDDLCommandSuite Author: Dilip Biswal <[email protected]> Closes apache#15190 from dilipbiswal/orc.

…eating Hive Serde Tables ## What changes were proposed in this pull request? Reopens the closed PR apache#15190 (Please refer to the above link for review comments on the PR) Make sure the hive.default.fileformat is used to when creating the storage format metadata. Output ``` SQL scala> spark.sql("SET hive.default.fileformat=orc") res1: org.apache.spark.sql.DataFrame = [key: string, value: string] scala> spark.sql("CREATE TABLE tmp_default(id INT)") res2: org.apache.spark.sql.DataFrame = [] ``` Before ```SQL scala> spark.sql("DESC FORMATTED tmp_default").collect.foreach(println) .. [# Storage Information,,] [SerDe Library:,org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe,] [InputFormat:,org.apache.hadoop.hive.ql.io.orc.OrcInputFormat,] [OutputFormat:,org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat,] [Compressed:,No,] [Storage Desc Parameters:,,] [ serialization.format,1,] ``` After ```SQL scala> spark.sql("DESC FORMATTED tmp_default").collect.foreach(println) .. [# Storage Information,,] [SerDe Library:,org.apache.hadoop.hive.ql.io.orc.OrcSerde,] [InputFormat:,org.apache.hadoop.hive.ql.io.orc.OrcInputFormat,] [OutputFormat:,org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat,] [Compressed:,No,] [Storage Desc Parameters:,,] [ serialization.format,1,] ``` ## How was this patch tested? (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) Added new tests to HiveDDLCommandSuite, SQLQuerySuite Author: Dilip Biswal <[email protected]> Closes apache#15495 from dilipbiswal/orc2.

…eating Hive Serde Tables ## What changes were proposed in this pull request? Make sure the hive.default.fileformat is used to when creating the storage format metadata. Output ``` SQL scala> spark.sql("SET hive.default.fileformat=orc") res1: org.apache.spark.sql.DataFrame = [key: string, value: string] scala> spark.sql("CREATE TABLE tmp_default(id INT)") res2: org.apache.spark.sql.DataFrame = [] ``` Before ```SQL scala> spark.sql("DESC FORMATTED tmp_default").collect.foreach(println) .. [# Storage Information,,] [SerDe Library:,org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe,] [InputFormat:,org.apache.hadoop.hive.ql.io.orc.OrcInputFormat,] [OutputFormat:,org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat,] [Compressed:,No,] [Storage Desc Parameters:,,] [ serialization.format,1,] ``` After ```SQL scala> spark.sql("DESC FORMATTED tmp_default").collect.foreach(println) .. [# Storage Information,,] [SerDe Library:,org.apache.hadoop.hive.ql.io.orc.OrcSerde,] [InputFormat:,org.apache.hadoop.hive.ql.io.orc.OrcInputFormat,] [OutputFormat:,org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat,] [Compressed:,No,] [Storage Desc Parameters:,,] [ serialization.format,1,] ``` ## How was this patch tested? Added new tests to HiveDDLCommandSuite Author: Dilip Biswal <[email protected]> Closes apache#15190 from dilipbiswal/orc.

…eating Hive Serde Tables ## What changes were proposed in this pull request? Reopens the closed PR apache#15190 (Please refer to the above link for review comments on the PR) Make sure the hive.default.fileformat is used to when creating the storage format metadata. Output ``` SQL scala> spark.sql("SET hive.default.fileformat=orc") res1: org.apache.spark.sql.DataFrame = [key: string, value: string] scala> spark.sql("CREATE TABLE tmp_default(id INT)") res2: org.apache.spark.sql.DataFrame = [] ``` Before ```SQL scala> spark.sql("DESC FORMATTED tmp_default").collect.foreach(println) .. [# Storage Information,,] [SerDe Library:,org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe,] [InputFormat:,org.apache.hadoop.hive.ql.io.orc.OrcInputFormat,] [OutputFormat:,org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat,] [Compressed:,No,] [Storage Desc Parameters:,,] [ serialization.format,1,] ``` After ```SQL scala> spark.sql("DESC FORMATTED tmp_default").collect.foreach(println) .. [# Storage Information,,] [SerDe Library:,org.apache.hadoop.hive.ql.io.orc.OrcSerde,] [InputFormat:,org.apache.hadoop.hive.ql.io.orc.OrcInputFormat,] [OutputFormat:,org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat,] [Compressed:,No,] [Storage Desc Parameters:,,] [ serialization.format,1,] ``` ## How was this patch tested? (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) Added new tests to HiveDDLCommandSuite, SQLQuerySuite Author: Dilip Biswal <[email protected]> Closes apache#15495 from dilipbiswal/orc2.

dilipbiswal added 2 commits September 21, 2016 15:50

[SPARK-17620] hive.default.fileformat=orc does not set OrcSerde

042a94e

style

f60e760

gatorsmile reviewed Sep 22, 2016

View reviewed changes

review

f2b93de

dilipbiswal changed the title ~~[SPARK-17620][SQL] hive.default.fileformat=orc does not set OrcSerde~~ [SPARK-17620][SQL] Use the storage format specified by hive.default.fileformat when creating hive serde tables. Sep 22, 2016

viirya reviewed Sep 22, 2016

View reviewed changes

dilipbiswal changed the title ~~[SPARK-17620][SQL] Use the storage format specified by hive.default.fileformat when creating hive serde tables.~~ [SPARK-17620][SQL] Determine Serde by hive.default.fileformat when Creating Hive Serde Tables Sep 22, 2016

viirya reviewed Sep 22, 2016

View reviewed changes

Add test

f32fe21

asfgit closed this in 7ab8624 Oct 14, 2016

dilipbiswal mentioned this pull request Oct 14, 2016

[SPARK-17620][SQL] Determine Serde by hive.default.fileformat when Creating Hive Serde Tables #15495

Closed

[SPARK-17620][SQL] Determine Serde by hive.default.fileformat when Creating Hive Serde Tables #15190

[SPARK-17620][SQL] Determine Serde by hive.default.fileformat when Creating Hive Serde Tables #15190

Uh oh!

Conversation

dilipbiswal commented Sep 21, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented Sep 22, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gatorsmile commented Sep 22, 2016

Uh oh!

SparkQA commented Sep 22, 2016

Uh oh!

SparkQA commented Sep 22, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gatorsmile Sep 22, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gatorsmile commented Sep 22, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

viirya commented Sep 22, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dilipbiswal commented Sep 22, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

viirya commented Sep 22, 2016

Uh oh!

viirya commented Sep 22, 2016

Uh oh!

dilipbiswal commented Sep 22, 2016

Uh oh!

viirya commented Sep 22, 2016

Uh oh!

viirya commented Sep 22, 2016

Uh oh!

viirya commented Sep 22, 2016

Uh oh!

yhuai commented Sep 22, 2016

Uh oh!

dilipbiswal commented Sep 22, 2016

Uh oh!

gatorsmile commented Sep 22, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dilipbiswal commented Sep 22, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gatorsmile commented Sep 22, 2016

Uh oh!

viirya commented Sep 23, 2016

Uh oh!

gatorsmile commented Oct 11, 2016

dilipbiswal commented Sep 21, 2016 •

edited

Loading

gatorsmile Sep 22, 2016 •

edited

Loading

viirya commented Sep 22, 2016 •

edited

Loading

dilipbiswal commented Sep 22, 2016 •

edited

Loading

gatorsmile commented Sep 22, 2016 •

edited

Loading

dilipbiswal commented Sep 22, 2016 •

edited

Loading