Skip to content

[SPARK-25660][SQL] Fix for the backward slash as CSV fields delimiter #22654

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
MaxGekk wants to merge 6 commits into from
Closed

[SPARK-25660][SQL] Fix for the backward slash as CSV fields delimiter #22654

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
dd16ca3
Test for backward slash as the delimiter
MaxGekk Oct 6, 2018
7bf453a
Bug fix + tests
MaxGekk Oct 6, 2018
728aac2
Removing string interpolation
MaxGekk Oct 9, 2018
983315b
Merge remote-tracking branch 'origin/master' into csv-slash-delim
MaxGekk Oct 9, 2018
1c2ac25
Support backslash escaped by backslash
MaxGekk Oct 9, 2018
20856b4
Prohibit single backslash
MaxGekk Oct 12, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 15 additions & 17 deletions sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVUtils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,23 +97,21 @@ object CSVUtils {
*/
@throws[IllegalArgumentException]
def toChar(str: String): Char = {
if (str.charAt(0) == '\\') {
str.charAt(1)
match {
case 't' => '\t'
case 'r' => '\r'
case 'b' => '\b'
case 'f' => '\f'
case '\"' => '\"' // In case user changes quote char and uses \" as delimiter in options
case '\'' => '\''
case 'u' if str == """\u0000""" => '\u0000'
case _ =>
throw new IllegalArgumentException(s"Unsupported special character for delimiter: $str")
}
} else if (str.length == 1) {
str.charAt(0)
} else {
throw new IllegalArgumentException(s"Delimiter cannot be more than one character: $str")
(str: Seq[Char]) match {
case Seq() => throw new IllegalArgumentException(s"Delimiter cannot be empty string")
case Seq(c) => c
Copy link
Member

@srowen srowen Oct 9, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm missing why we had to switch up the case statement like this. I get that we need to cover more cases, but there was duplication and now there is a bit more. What about ...

str.length match {
  case 0 => // error
  case 1 => str(0)
  case 2 if str(0) == '\\' =>
    str(1) match {
      case c if """trbf"'\""".contains(c) => c
      case 'u' if str == """\u0000""" => '\0'
      case _ => // error
    }
  case _ => // error
}

Copy link
Member Author

@MaxGekk MaxGekk Oct 9, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer more declarative way and less nested levels of controls. but this is personal opinion. Let's look at your example:

str.length

you didn't check that str can be null.

case 2 if str(0) == '\\' =>
case 'u' if str == """\u0000""" => '\0'

If it has length 2, how str could be """\u0000"""?

case c if """trbf"'\""".contains(c) => c

You should produce control chars not just second char. For example: \t -> Seq('', 't') -> '\t`.

In my approach, everything is simple. One input case is mapped to one output. There is no unnecessary complexity.

Copy link
Member

@srowen srowen Oct 9, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yeah good points. This is too clever, off the top of my head. I still wonder if the code here can reduce the duplication of Seq('\\', c) => '\c' but I don't see a way that actually works, yeah.

case Seq('\\', 't') => '\t'
case Seq('\\', 'r') => '\r'
case Seq('\\', 'b') => '\b'
case Seq('\\', 'f') => '\f'
// In case user changes quote char and uses \" as delimiter in options
case Seq('\\', '\"') => '\"'
case Seq('\\', '\'') => '\''
case _ if str == """\u0000""" => '\u0000'
case Seq('\\', _) =>
throw new IllegalArgumentException(s"Unsupported special character for delimiter: $str")
case _ =>
throw new IllegalArgumentException(s"Delimiter cannot be more than one character: $str")
}
}

Expand Down
9 changes: 9 additions & 0 deletions sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1820,4 +1820,13 @@ class CSVSuite extends QueryTest with SharedSQLContext with SQLTestUtils with Te
checkAnswer(spark.read.option("multiLine", true).schema(schema).csv(input), Row(null))
assert(spark.read.csv(input).collect().toSet == Set(Row()))
}

test("using the backward slash as the delimiter") {
val input = Seq("""abc\1""").toDS()
Copy link
Member

@srowen srowen Oct 9, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't \ the default escape character? this should be read as the string "abc1" then, and not delimited. It would have to be \\, right? I'm not talking about Scala string escaping, but CSV here.

Or is the point that delimiting takes precedence?

Copy link
Member Author

@MaxGekk MaxGekk Oct 9, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or is the point that delimiting takes precedence?

Right, if an user specified \ as a delimiter, CSV parser considers it as the delimiter first of all. We can see that in the main loop that delimiters are handled before escape characters:
https://github.com/uniVocity/univocity-parsers/blob/6746adc2ddb420ebba7441339887e4bbc35cf087/src/main/java/com/univocity/parsers/csv/CsvParser.java#L115

Copy link
Member

@gatorsmile gatorsmile Oct 11, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If a user specified \ as a delimiter, we should issue an exception message and let users add the escape symbol i.e., \\. It is weird to see both \\ and \ are representing the same thing \. We should be consistent for handling backslash in all the cases.

Copy link
Member Author

@MaxGekk MaxGekk Oct 12, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prohibited single backslash and throw an exception with a tip of using double backslash.

checkAnswer(spark.read.option("delimiter", "\\").csv(input), Row("abc", "1"))
checkAnswer(spark.read.option("inferSchema", true).option("delimiter", "\\").csv(input),
Row("abc", 1))
val schema = new StructType().add("a", StringType).add("b", IntegerType)
checkAnswer(spark.read.schema(schema).option("delimiter", "\\").csv(input), Row("abc", 1))
}
}
10 changes: 10 additions & 0 deletions sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVUtilsSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,4 +44,14 @@ class CSVUtilsSuite extends SparkFunSuite {
assert(exception.getMessage.contains("Unsupported special character for delimiter"))
}

test("string with one backward slash should be converted to the backward slash char") {
assert(CSVUtils.toChar("""\""") == '\\')
}

test("output proper error message for empty string") {
val exception = intercept[IllegalArgumentException]{
CSVUtils.toChar("")
}
assert(exception.getMessage.contains("Delimiter cannot be empty string"))
}
}