Skip to content

Commit 270916f

Browse files
dongjoon-hyundongjoon-hyun
committed
[SPARK-26584][SQL] Remove spark.sql.orc.copyBatchToSpark internal conf
## What changes were proposed in this pull request? This PR aims to remove internal ORC configuration to simplify the code path for Spark 3.0.0. This removes the configuration `spark.sql.orc.copyBatchToSpark` and related ORC codes including tests and benchmarks. ## How was this patch tested? Pass the Jenkins with the reduced test coverage. Closes #23503 from dongjoon-hyun/SPARK-26584. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
1 parent 2f8a938 commit 270916f

8 files changed

Lines changed: 26 additions & 523 deletions

File tree

‎sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala‎

Lines changed: 0 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -524,13 +524,6 @@ object SQLConf {
524524
.intConf
525525
.createWithDefault(4096)
526526

527-
val ORC_COPY_BATCH_TO_SPARK = buildConf("spark.sql.orc.copyBatchToSpark")
528-
.doc("Whether or not to copy the ORC columnar batch to Spark columnar batch in the " +
529-
"vectorized ORC reader.")
530-
.internal()
531-
.booleanConf
532-
.createWithDefault(false)
533-
534527
val ORC_FILTER_PUSHDOWN_ENABLED = buildConf("spark.sql.orc.filterPushdown")
535528
.doc("When true, enable filter pushdown for ORC files.")
536529
.booleanConf

‎sql/core/benchmarks/DataSourceReadBenchmark-results.txt‎

Lines changed: 0 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,6 @@ SQL Json 8709 / 8724 1.8 5
1111
SQL Parquet Vectorized 166 / 187 94.8 10.5 159.0X
1212
SQL Parquet MR 1706 / 1720 9.2 108.4 15.5X
1313
SQL ORC Vectorized 167 / 174 94.2 10.6 157.9X
14-
SQL ORC Vectorized with copy 226 / 231 69.6 14.4 116.7X
1514
SQL ORC MR 1433 / 1465 11.0 91.1 18.4X
1615

1716
OpenJDK 64-Bit Server VM 1.8.0_191-b12 on Linux 3.10.0-862.3.2.el7.x86_64
@@ -30,7 +29,6 @@ SQL Json 8990 / 8998 1.7 5
3029
SQL Parquet Vectorized 209 / 221 75.1 13.3 126.5X
3130
SQL Parquet MR 1949 / 1949 8.1 123.9 13.6X
3231
SQL ORC Vectorized 221 / 228 71.3 14.0 120.1X
33-
SQL ORC Vectorized with copy 315 / 319 49.9 20.1 84.0X
3432
SQL ORC MR 1527 / 1549 10.3 97.1 17.3X
3533

3634
OpenJDK 64-Bit Server VM 1.8.0_191-b12 on Linux 3.10.0-862.3.2.el7.x86_64
@@ -49,7 +47,6 @@ SQL Json 9703 / 9733 1.6 6
4947
SQL Parquet Vectorized 176 / 182 89.2 11.2 157.0X
5048
SQL Parquet MR 2164 / 2173 7.3 137.6 12.8X
5149
SQL ORC Vectorized 307 / 314 51.2 19.5 90.2X
52-
SQL ORC Vectorized with copy 312 / 319 50.4 19.8 88.7X
5350
SQL ORC MR 1690 / 1700 9.3 107.4 16.4X
5451

5552
OpenJDK 64-Bit Server VM 1.8.0_191-b12 on Linux 3.10.0-862.3.2.el7.x86_64
@@ -68,7 +65,6 @@ SQL Json 12570 / 12617 1.3 7
6865
SQL Parquet Vectorized 270 / 308 58.2 17.2 128.9X
6966
SQL Parquet MR 2427 / 2431 6.5 154.3 14.3X
7067
SQL ORC Vectorized 388 / 398 40.6 24.6 89.8X
71-
SQL ORC Vectorized with copy 395 / 402 39.9 25.1 88.2X
7268
SQL ORC MR 1819 / 1851 8.6 115.7 19.1X
7369

7470
OpenJDK 64-Bit Server VM 1.8.0_191-b12 on Linux 3.10.0-862.3.2.el7.x86_64
@@ -87,7 +83,6 @@ SQL Json 12039 / 12215 1.3 7
8783
SQL Parquet Vectorized 170 / 177 92.4 10.8 169.0X
8884
SQL Parquet MR 2184 / 2196 7.2 138.9 13.2X
8985
SQL ORC Vectorized 432 / 440 36.4 27.5 66.5X
90-
SQL ORC Vectorized with copy 439 / 442 35.9 27.9 65.6X
9186
SQL ORC MR 1812 / 1833 8.7 115.2 15.9X
9287

9388
OpenJDK 64-Bit Server VM 1.8.0_191-b12 on Linux 3.10.0-862.3.2.el7.x86_64
@@ -106,7 +101,6 @@ SQL Json 18895 / 18898 0.8 12
106101
SQL Parquet Vectorized 267 / 276 58.9 17.0 135.6X
107102
SQL Parquet MR 2355 / 2363 6.7 149.7 15.4X
108103
SQL ORC Vectorized 543 / 546 29.0 34.5 66.6X
109-
SQL ORC Vectorized with copy 548 / 557 28.7 34.8 66.0X
110104
SQL ORC MR 2246 / 2258 7.0 142.8 16.1X
111105

112106
OpenJDK 64-Bit Server VM 1.8.0_191-b12 on Linux 3.10.0-862.3.2.el7.x86_64
@@ -130,7 +124,6 @@ SQL Json 12145 / 12174 0.9 11
130124
SQL Parquet Vectorized 2363 / 2377 4.4 225.3 8.9X
131125
SQL Parquet MR 4555 / 4557 2.3 434.4 4.6X
132126
SQL ORC Vectorized 2361 / 2388 4.4 225.1 9.0X
133-
SQL ORC Vectorized with copy 2540 / 2557 4.1 242.2 8.3X
134127
SQL ORC MR 4186 / 4209 2.5 399.2 5.0X
135128

136129

@@ -147,7 +140,6 @@ SQL Json 7025 / 7025 1.5 6
147140
SQL Parquet Vectorized 803 / 821 13.1 76.6 14.6X
148141
SQL Parquet MR 1776 / 1790 5.9 169.4 6.6X
149142
SQL ORC Vectorized 491 / 494 21.4 46.8 23.8X
150-
SQL ORC Vectorized with copy 723 / 725 14.5 68.9 16.2X
151143
SQL ORC MR 2050 / 2063 5.1 195.5 5.7X
152144

153145

@@ -164,21 +156,18 @@ Data column - Json 12876 / 12882 1.2 8
164156
Data column - Parquet Vectorized 277 / 282 56.7 17.6 111.6X
165157
Data column - Parquet MR 3398 / 3402 4.6 216.0 9.1X
166158
Data column - ORC Vectorized 399 / 407 39.4 25.4 77.5X
167-
Data column - ORC Vectorized with copy 407 / 447 38.6 25.9 76.0X
168159
Data column - ORC MR 2583 / 2589 6.1 164.2 12.0X
169160
Partition column - CSV 7403 / 7427 2.1 470.7 4.2X
170161
Partition column - Json 5587 / 5625 2.8 355.2 5.5X
171162
Partition column - Parquet Vectorized 71 / 78 222.6 4.5 438.3X
172163
Partition column - Parquet MR 1798 / 1808 8.7 114.3 17.2X
173164
Partition column - ORC Vectorized 72 / 75 219.0 4.6 431.2X
174-
Partition column - ORC Vectorized with copy 71 / 77 221.1 4.5 435.4X
175165
Partition column - ORC MR 1772 / 1778 8.9 112.6 17.5X
176166
Both columns - CSV 30211 / 30212 0.5 1920.7 1.0X
177167
Both columns - Json 13382 / 13391 1.2 850.8 2.3X
178168
Both columns - Parquet Vectorized 321 / 333 49.0 20.4 96.4X
179169
Both columns - Parquet MR 3656 / 3661 4.3 232.4 8.5X
180170
Both columns - ORC Vectorized 443 / 448 35.5 28.2 69.9X
181-
Both column - ORC Vectorized with copy 527 / 533 29.9 33.5 58.8X
182171
Both columns - ORC MR 2626 / 2633 6.0 167.0 11.8X
183172

184173

@@ -196,7 +185,6 @@ SQL Parquet Vectorized 1563 / 1564 6.7 1
196185
SQL Parquet MR 3835 / 3836 2.7 365.8 3.6X
197186
ParquetReader Vectorized 1115 / 1118 9.4 106.4 12.5X
198187
SQL ORC Vectorized 1172 / 1208 8.9 111.8 11.9X
199-
SQL ORC Vectorized with copy 1630 / 1644 6.4 155.5 8.5X
200188
SQL ORC MR 3708 / 3711 2.8 353.6 3.8X
201189

202190
OpenJDK 64-Bit Server VM 1.8.0_191-b12 on Linux 3.10.0-862.3.2.el7.x86_64
@@ -209,7 +197,6 @@ SQL Parquet Vectorized 1103 / 1112 9.5 1
209197
SQL Parquet MR 2841 / 2847 3.7 271.0 4.9X
210198
ParquetReader Vectorized 992 / 1012 10.6 94.6 14.1X
211199
SQL ORC Vectorized 1275 / 1349 8.2 121.6 11.0X
212-
SQL ORC Vectorized with copy 1631 / 1644 6.4 155.5 8.6X
213200
SQL ORC MR 3244 / 3259 3.2 309.3 4.3X
214201

215202
OpenJDK 64-Bit Server VM 1.8.0_191-b12 on Linux 3.10.0-862.3.2.el7.x86_64
@@ -222,7 +209,6 @@ SQL Parquet Vectorized 238 / 242 44.1
222209
SQL Parquet MR 1730 / 1734 6.1 165.0 6.5X
223210
ParquetReader Vectorized 237 / 238 44.3 22.6 47.4X
224211
SQL ORC Vectorized 459 / 462 22.8 43.8 24.4X
225-
SQL ORC Vectorized with copy 581 / 583 18.1 55.4 19.3X
226212
SQL ORC MR 1767 / 1783 5.9 168.5 6.4X
227213

228214

@@ -239,7 +225,6 @@ SQL Json 2808 / 2843 0.4 26
239225
SQL Parquet Vectorized 56 / 63 18.9 52.9 59.8X
240226
SQL Parquet MR 215 / 219 4.9 205.4 15.4X
241227
SQL ORC Vectorized 64 / 76 16.4 60.9 52.0X
242-
SQL ORC Vectorized with copy 64 / 67 16.3 61.3 51.7X
243228
SQL ORC MR 314 / 316 3.3 299.6 10.6X
244229

245230
OpenJDK 64-Bit Server VM 1.8.0_191-b12 on Linux 3.10.0-862.3.2.el7.x86_64
@@ -251,7 +236,6 @@ SQL Json 10294 / 10325 0.1 98
251236
SQL Parquet Vectorized 72 / 85 14.5 69.0 110.3X
252237
SQL Parquet MR 237 / 241 4.4 226.4 33.6X
253238
SQL ORC Vectorized 82 / 92 12.7 78.5 97.0X
254-
SQL ORC Vectorized with copy 82 / 88 12.7 78.5 97.0X
255239
SQL ORC MR 900 / 909 1.2 858.5 8.9X
256240

257241
OpenJDK 64-Bit Server VM 1.8.0_191-b12 on Linux 3.10.0-862.3.2.el7.x86_64
@@ -263,7 +247,6 @@ SQL Json 18813 / 18827 0.1 179
263247
SQL Parquet Vectorized 107 / 111 9.8 101.8 126.3X
264248
SQL Parquet MR 275 / 286 3.8 262.3 49.0X
265249
SQL ORC Vectorized 107 / 115 9.8 101.7 126.4X
266-
SQL ORC Vectorized with copy 107 / 115 9.8 102.3 125.8X
267250
SQL ORC MR 1659 / 1664 0.6 1582.3 8.1X
268251

269252

0 commit comments

Comments
 (0)