Skip to content

Conversation

@MaxGekk
Copy link
Member

@MaxGekk MaxGekk commented Jul 23, 2020

What changes were proposed in this pull request?

Refactoring of JsonFilters:

  • Add an assert to the skipRow method to check the input index
  • Move checking of the SQL config spark.sql.json.filterPushdown.enabled from JsonFilters to JacksonParser.

Why are the changes needed?

  1. The assert should catch incorrect usage of JsonFilters
  2. The config checking out of JsonFilters makes it consistent with OrderedFilters (see [SPARK-32346][SQL] Support filters pushdown in Avro datasource #29145).
  3. JsonFilters can be used by other datasource in the future and don't depend from the JSON configs.

Does this PR introduce any user-facing change?

No

How was this patch tested?

By existing tests suites:

$ build/sbt "sql/test:testOnly org.apache.spark.sql.execution.datasources.json.*"
$ build/sbt "test:testOnly org.apache.spark.sql.catalyst.json.*"

@SparkQA
Copy link

SparkQA commented Jul 23, 2020

Test build #126418 has finished for PR 29206 at commit 8e3b94d.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon
Copy link
Member

Merged to master.

@MaxGekk MaxGekk deleted the json-filters-pushdown-followup branch December 11, 2020 20:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants