[SPARK-16320][SPARK-16321] Fixing performance regression when reading Parquet

Maciej Brynski · Maciej Brynski · commit 652f557c1e4b · 2016-08-02T20:01:12.000Z
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala
@@ -325,6 +325,8 @@ private[sql] class ParquetFileFormat
         None
       }
 
+    pushed.foreach(ParquetInputFormat.setFilterPredicate(hadoopConf, _))
+
     val broadcastedHadoopConf =
       sparkSession.sparkContext.broadcast(new SerializableConfiguration(hadoopConf))
 

Original file line number	Diff line number	Diff line change
`@@ -325,6 +325,8 @@ private[sql] class ParquetFileFormat`
`325`	`325`	`None`
`326`	`326`	`}`
`327`	`327`
	`328`	`+ pushed.foreach(ParquetInputFormat.setFilterPredicate(hadoopConf, _))`
	`329`	`+`
`328`	`330`	`val broadcastedHadoopConf =`
`329`	`331`	`sparkSession.sparkContext.broadcast(new SerializableConfiguration(hadoopConf))`
`330`	`332`