Skip to content

[SPARK-34434][SQL] Mention DS rebase options in SparkUpgradeException #31562

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
MaxGekk wants to merge 1 commit into from
Closed

[SPARK-34434][SQL] Mention DS rebase options in SparkUpgradeException #31562

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 12 additions & 7 deletions sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceUtils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ import org.apache.spark.sql.{SPARK_LEGACY_DATETIME, SPARK_LEGACY_INT96, SPARK_VE
import org.apache.spark.sql.AnalysisException
import org.apache.spark.sql.catalyst.catalog.{CatalogTable, CatalogUtils}
import org.apache.spark.sql.catalyst.util.RebaseDateTime
import org.apache.spark.sql.execution.datasources.parquet.ParquetOptions
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.internal.SQLConf.LegacyBehaviorPolicy
import org.apache.spark.sql.sources.BaseRelation
Expand Down Expand Up @@ -132,19 +133,23 @@ object DataSourceUtils {
}

def newRebaseExceptionInRead(format: String): SparkUpgradeException = {
val config = format match {
case "Parquet INT96" => SQLConf.LEGACY_PARQUET_INT96_REBASE_MODE_IN_READ.key
case "Parquet" => SQLConf.LEGACY_PARQUET_REBASE_MODE_IN_READ.key
case "Avro" => SQLConf.LEGACY_AVRO_REBASE_MODE_IN_READ.key
val (config, option) = format match {
case "Parquet INT96" =>
(SQLConf.LEGACY_PARQUET_INT96_REBASE_MODE_IN_READ.key, ParquetOptions.INT96_REBASE_MODE)
case "Parquet" =>
(SQLConf.LEGACY_PARQUET_REBASE_MODE_IN_READ.key, ParquetOptions.DATETIME_REBASE_MODE)
case "Avro" =>
(SQLConf.LEGACY_AVRO_REBASE_MODE_IN_READ.key, "datetimeRebaseMode")
Copy link
Member

@dongjoon-hyun dongjoon-hyun Feb 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that this is inevitable because AvroOptions lives in external module.

case _ => throw new IllegalStateException("unrecognized format " + format)
}
new SparkUpgradeException("3.0", "reading dates before 1582-10-15 or timestamps before " +
s"1900-01-01T00:00:00Z from $format files can be ambiguous, as the files may be written by " +
"Spark 2.x or legacy versions of Hive, which uses a legacy hybrid calendar that is " +
"different from Spark 3.0+'s Proleptic Gregorian calendar. See more details in " +
s"SPARK-31404. You can set $config to 'LEGACY' to rebase the datetime values w.r.t. " +
s"the calendar difference during reading. Or set $config to 'CORRECTED' to read the " +
"datetime values as it is.", null)
s"SPARK-31404. You can set the SQL config '$config' or the datasource option '$option' to " +
"'LEGACY' to rebase the datetime values w.r.t. the calendar difference during reading. " +
s"To read the datetime values as it is, set the SQL config '$config' or " +
s"the datasource option '$option' to 'CORRECTED'.", null)
}

def newRebaseExceptionInWrite(format: String): SparkUpgradeException = {
Expand Down