Skip to content

Commit 6ed93c3

Browse files
GuoPhilipsemaropu
authored andcommitted
[SPARK-31753][SQL][DOCS] Add missing keywords in the SQL docs
### What changes were proposed in this pull request? update sql-ref docs, the following key words will be added in this PR. CASE/ELSE WHEN/THEN MAP KEYS TERMINATED BY NULL DEFINED AS LINES TERMINATED BY ESCAPED BY COLLECTION ITEMS TERMINATED BY PIVOT LATERAL VIEW OUTER? ROW FORMAT SERDE ROW FORMAT DELIMITED FIELDS TERMINATED BY IGNORE NULLS FIRST LAST ### Why are the changes needed? let more users know the sql key words usage ### Does this PR introduce _any_ user-facing change? ![image](https://user-images.githubusercontent.com/46367746/88148830-c6dc1f80-cc31-11ea-81ea-13bc9dc34550.png) ![image](https://user-images.githubusercontent.com/46367746/88148968-fb4fdb80-cc31-11ea-8649-e8297cf5813e.png) ![image](https://user-images.githubusercontent.com/46367746/88149000-073b9d80-cc32-11ea-9aa4-f914ecd72663.png) ![image](https://user-images.githubusercontent.com/46367746/88149021-0f93d880-cc32-11ea-86ed-7db8672b5aac.png) ### How was this patch tested? No Closes #29056 from GuoPhilipse/add-missing-keywords. Lead-authored-by: GuoPhilipse <guofei_ok@126.com> Co-authored-by: GuoPhilipse <46367746+GuoPhilipse@users.noreply.github.com> Signed-off-by: Takeshi Yamamuro <yamamuro@apache.org> (cherry picked from commit 8de4333) Signed-off-by: Takeshi Yamamuro <yamamuro@apache.org>
1 parent 4b8761e commit 6ed93c3

16 files changed

Lines changed: 520 additions & 25 deletions

docs/_data/menu-sql.yaml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -187,6 +187,12 @@
187187
url: sql-ref-syntax-qry-select-tvf.html
188188
- text: Window Function
189189
url: sql-ref-syntax-qry-select-window.html
190+
- text: CASE Clause
191+
url: sql-ref-syntax-qry-select-case.html
192+
- text: LATERAL VIEW Clause
193+
url: sql-ref-syntax-qry-select-lateral-view.html
194+
- text: PIVOT Clause
195+
url: sql-ref-syntax-qry-select-pivot.html
190196
- text: EXPLAIN
191197
url: sql-ref-syntax-qry-explain.html
192198
- text: Auxiliary Statements

docs/sql-ref-syntax-ddl-create-table-hiveformat.md

Lines changed: 90 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,14 @@ CREATE [ EXTERNAL ] TABLE [ IF NOT EXISTS ] table_identifier
3636
[ LOCATION path ]
3737
[ TBLPROPERTIES ( key1=val1, key2=val2, ... ) ]
3838
[ AS select_statement ]
39+
40+
row_format:
41+
: SERDE serde_class [ WITH SERDEPROPERTIES (k1=v1, k2=v2, ... ) ]
42+
| DELIMITED [ FIELDS TERMINATED BY fields_termiated_char [ ESCAPED BY escaped_char ] ]
43+
[ COLLECTION ITEMS TERMINATED BY collection_items_termiated_char ]
44+
[ MAP KEYS TERMINATED BY map_key_termiated_char ]
45+
[ LINES TERMINATED BY row_termiated_char ]
46+
[ NULL DEFINED AS null_char ]
3947
```
4048

4149
Note that, the clauses between the columns definition clause and the AS SELECT clause can come in
@@ -51,15 +59,55 @@ as any order. For example, you can write COMMENT table_comment after TBLPROPERTI
5159

5260
* **EXTERNAL**
5361

54-
Table is defined using the path provided as LOCATION, does not use default location for this table.
62+
Table is defined using the path provided as `LOCATION`, does not use default location for this table.
5563

5664
* **PARTITIONED BY**
5765

5866
Partitions are created on the table, based on the columns specified.
67+
68+
* **row_format**
69+
70+
Use the `SERDE` clause to specify a custom SerDe for one table. Otherwise, use the `DELIMITED` clause to use the native SerDe and specify the delimiter, escape character, null character and so on.
71+
72+
* **SERDE**
73+
74+
Specifies a custom SerDe for one table.
75+
76+
* **serde_class**
77+
78+
Specifies a fully-qualified class name of a custom SerDe.
79+
80+
* **SERDEPROPERTIES**
81+
82+
A list of key-value pairs that is used to tag the SerDe definition.
83+
84+
* **DELIMITED**
85+
86+
The `DELIMITED` clause can be used to specify the native SerDe and state the delimiter, escape character, null character and so on.
87+
88+
* **FIELDS TERMINATED BY**
5989

60-
* **ROW FORMAT**
90+
Used to define a column separator.
91+
92+
* **COLLECTION ITEMS TERMINATED BY**
6193

62-
SERDE is used to specify a custom SerDe or the DELIMITED clause in order to use the native SerDe.
94+
Used to define a collection item separator.
95+
96+
* **MAP KEYS TERMINATED BY**
97+
98+
Used to define a map key separator.
99+
100+
* **LINES TERMINATED BY**
101+
102+
Used to define a row separator.
103+
104+
* **NULL DEFINED AS**
105+
106+
Used to define the specific value for NULL.
107+
108+
* **ESCAPED BY**
109+
110+
Used for escape mechanism.
63111

64112
* **STORED AS**
65113

@@ -114,9 +162,47 @@ CREATE TABLE student (id INT, name STRING)
114162
PARTITIONED BY (age INT);
115163

116164
--Use Row Format and file format
117-
CREATE TABLE student (id INT,name STRING)
165+
CREATE TABLE student (id INT, name STRING)
118166
ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
119167
STORED AS TEXTFILE;
168+
169+
--Use complex datatype
170+
CREATE EXTERNAL TABLE family(
171+
name STRING,
172+
friends ARRAY<STRING>,
173+
children MAP<STRING, INT>,
174+
address STRUCT<street: STRING, city: STRING>
175+
)
176+
ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' ESCAPED BY '\\'
177+
COLLECTION ITEMS TERMINATED BY '_'
178+
MAP KEYS TERMINATED BY ':'
179+
LINES TERMINATED BY '\n'
180+
NULL DEFINED AS 'foonull'
181+
STORED AS TEXTFILE
182+
LOCATION '/tmp/family/';
183+
184+
--Use predefined custom SerDe
185+
CREATE TABLE avroExample
186+
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
187+
STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
188+
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
189+
TBLPROPERTIES ('avro.schema.literal'='{ "namespace": "org.apache.hive",
190+
"name": "first_schema",
191+
"type": "record",
192+
"fields": [
193+
{ "name":"string1", "type":"string" },
194+
{ "name":"string2", "type":"string" }
195+
] }');
196+
197+
--Use personalized custom SerDe(we may need to `ADD JAR xxx.jar` first to ensure we can find the serde_class,
198+
--or you may run into `CLASSNOTFOUND` exception)
199+
ADD JAR /tmp/hive_serde_example.jar;
200+
201+
CREATE EXTERNAL TABLE family (id INT, name STRING)
202+
ROW FORMAT SERDE 'com.ly.spark.serde.SerDeExample'
203+
STORED AS INPUTFORMAT 'com.ly.spark.example.serde.io.SerDeExampleInputFormat'
204+
OUTPUTFORMAT 'com.ly.spark.example.serde.io.SerDeExampleOutputFormat'
205+
LOCATION '/tmp/family/';
120206
```
121207

122208
### Related Statements
Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
---
2+
layout: global
3+
title: CASE Clause
4+
displayTitle: CASE Clause
5+
license: |
6+
Licensed to the Apache Software Foundation (ASF) under one or more
7+
contributor license agreements. See the NOTICE file distributed with
8+
this work for additional information regarding copyright ownership.
9+
The ASF licenses this file to You under the Apache License, Version 2.0
10+
(the "License"); you may not use this file except in compliance with
11+
the License. You may obtain a copy of the License at
12+
13+
http://www.apache.org/licenses/LICENSE-2.0
14+
15+
Unless required by applicable law or agreed to in writing, software
16+
distributed under the License is distributed on an "AS IS" BASIS,
17+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
18+
See the License for the specific language governing permissions and
19+
limitations under the License.
20+
---
21+
22+
### Description
23+
24+
`CASE` clause uses a rule to return a specific result based on the specified condition, similar to if/else statements in other programming languages.
25+
26+
### Syntax
27+
28+
```sql
29+
CASE [ expression ] { WHEN boolean_expression THEN then_expression } [ ... ]
30+
[ ELSE else_expression ]
31+
END
32+
```
33+
34+
### Parameters
35+
36+
* **boolean_expression**
37+
38+
Specifies any expression that evaluates to a result type `boolean`. Two or
39+
more expressions may be combined together using the logical
40+
operators ( `AND`, `OR` ).
41+
42+
* **then_expression**
43+
44+
Specifies the then expression based on the `boolean_expression` condition; `then_expression` and `else_expression` should all be same type or coercible to a common type.
45+
46+
* **else_expression**
47+
48+
Specifies the default expression; `then_expression` and `else_expression` should all be same type or coercible to a common type.
49+
50+
### Examples
51+
52+
```sql
53+
CREATE TABLE person (id INT, name STRING, age INT);
54+
INSERT INTO person VALUES
55+
(100, 'John', 30),
56+
(200, 'Mary', NULL),
57+
(300, 'Mike', 80),
58+
(400, 'Dan', 50);
59+
60+
SELECT id, CASE WHEN id > 200 THEN 'bigger' ELSE 'small' END FROM person;
61+
+------+--------------------------------------------------+
62+
| id | CASE WHEN (id > 200) THEN bigger ELSE small END |
63+
+------+--------------------------------------------------+
64+
| 100 | small |
65+
| 200 | small |
66+
| 300 | bigger |
67+
| 400 | bigger |
68+
+------+--------------------------------------------------+
69+
70+
SELECT id, CASE id WHEN 100 then 'bigger' WHEN id > 300 THEN '300' ELSE 'small' END FROM person;
71+
+------+-----------------------------------------------------------------------------------------------+
72+
| id | CASE WHEN (id = 100) THEN bigger WHEN (id = CAST((id > 300) AS INT)) THEN 300 ELSE small END |
73+
+------+-----------------------------------------------------------------------------------------------+
74+
| 100 | bigger |
75+
| 200 | small |
76+
| 300 | small |
77+
| 400 | small |
78+
+------+-----------------------------------------------------------------------------------------------+
79+
80+
SELECT * FROM person
81+
WHERE
82+
CASE 1 = 1
83+
WHEN 100 THEN 'big'
84+
WHEN 200 THEN 'bigger'
85+
WHEN 300 THEN 'biggest'
86+
ELSE 'small'
87+
END = 'small';
88+
+------+-------+-------+
89+
| id | name | age |
90+
+------+-------+-------+
91+
| 100 | John | 30 |
92+
| 200 | Mary | NULL |
93+
| 300 | Mike | 80 |
94+
| 400 | Dan | 50 |
95+
+------+-------+-------+
96+
```
97+
98+
### Related Statements
99+
100+
* [SELECT Main](sql-ref-syntax-qry-select.html)
101+
* [WHERE Clause](sql-ref-syntax-qry-select-where.html)
102+
* [GROUP BY Clause](sql-ref-syntax-qry-select-groupby.html)
103+
* [HAVING Clause](sql-ref-syntax-qry-select-having.html)
104+
* [ORDER BY Clause](sql-ref-syntax-qry-select-orderby.html)
105+
* [SORT BY Clause](sql-ref-syntax-qry-select-sortby.html)
106+
* [DISTRIBUTE BY Clause](sql-ref-syntax-qry-select-distribute-by.html)
107+
* [LIMIT Clause](sql-ref-syntax-qry-select-limit.html)
108+
* [PIVOT Clause](sql-ref-syntax-qry-select-pivot.html)
109+
* [LATERAL VIEW Clause](sql-ref-syntax-qry-select-lateral-view.html)

docs/sql-ref-syntax-qry-select-clusterby.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -99,3 +99,6 @@ SELECT age, name FROM person CLUSTER BY age;
9999
* [SORT BY Clause](sql-ref-syntax-qry-select-sortby.html)
100100
* [DISTRIBUTE BY Clause](sql-ref-syntax-qry-select-distribute-by.html)
101101
* [LIMIT Clause](sql-ref-syntax-qry-select-limit.html)
102+
* [CASE Clause](sql-ref-syntax-qry-select-case.html)
103+
* [PIVOT Clause](sql-ref-syntax-qry-select-pivot.html)
104+
* [LATERAL VIEW Clause](sql-ref-syntax-qry-select-lateral-view.html)

docs/sql-ref-syntax-qry-select-distribute-by.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -94,3 +94,6 @@ SELECT age, name FROM person DISTRIBUTE BY age;
9494
* [SORT BY Clause](sql-ref-syntax-qry-select-sortby.html)
9595
* [CLUSTER BY Clause](sql-ref-syntax-qry-select-clusterby.html)
9696
* [LIMIT Clause](sql-ref-syntax-qry-select-limit.html)
97+
* [CASE Clause](sql-ref-syntax-qry-select-case.html)
98+
* [PIVOT Clause](sql-ref-syntax-qry-select-pivot.html)
99+
* [LATERAL VIEW Clause](sql-ref-syntax-qry-select-lateral-view.html)

docs/sql-ref-syntax-qry-select-groupby.md

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -260,6 +260,30 @@ SELECT city, car_model, sum(quantity) AS sum FROM dealer
260260
| San Jose| HondaAccord| 8|
261261
| San Jose| HondaCivic| 5|
262262
+---------+------------+---+
263+
264+
--Prepare data for ignore nulls example
265+
CREATE TABLE person (id INT, name STRING, age INT);
266+
INSERT INTO person VALUES
267+
(100, 'Mary', NULL),
268+
(200, 'John', 30),
269+
(300, 'Mike', 80),
270+
(400, 'Dan', 50);
271+
272+
--Select the first row in cloumn age
273+
SELECT FIRST(age) FROM person;
274+
+--------------------+
275+
| first(age, false) |
276+
+--------------------+
277+
| NULL |
278+
+--------------------+
279+
280+
--Get the first row in cloumn `age` ignore nulls,last row in column `id` and sum of cloumn `id`.
281+
SELECT FIRST(age IGNORE NULLS), LAST(id), SUM(id) FROM person;
282+
+-------------------+------------------+----------+
283+
| first(age, true) | last(id, false) | sum(id) |
284+
+-------------------+------------------+----------+
285+
| 30 | 400 | 1000 |
286+
+-------------------+------------------+----------+
263287
```
264288

265289
### Related Statements
@@ -272,3 +296,6 @@ SELECT city, car_model, sum(quantity) AS sum FROM dealer
272296
* [CLUSTER BY Clause](sql-ref-syntax-qry-select-clusterby.html)
273297
* [DISTRIBUTE BY Clause](sql-ref-syntax-qry-select-distribute-by.html)
274298
* [LIMIT Clause](sql-ref-syntax-qry-select-limit.html)
299+
* [CASE Clause](sql-ref-syntax-qry-select-case.html)
300+
* [PIVOT Clause](sql-ref-syntax-qry-select-pivot.html)
301+
* [LATERAL VIEW Clause](sql-ref-syntax-qry-select-lateral-view.html)

docs/sql-ref-syntax-qry-select-having.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -125,3 +125,6 @@ SELECT sum(quantity) AS sum FROM dealer HAVING sum(quantity) > 10;
125125
* [CLUSTER BY Clause](sql-ref-syntax-qry-select-clusterby.html)
126126
* [DISTRIBUTE BY Clause](sql-ref-syntax-qry-select-distribute-by.html)
127127
* [LIMIT Clause](sql-ref-syntax-qry-select-limit.html)
128+
* [CASE Clause](sql-ref-syntax-qry-select-case.html)
129+
* [PIVOT Clause](sql-ref-syntax-qry-select-pivot.html)
130+
* [LATERAL VIEW Clause](sql-ref-syntax-qry-select-lateral-view.html)

0 commit comments

Comments
 (0)