Skip to content

[FLINK-22119][hive][doc] Update document for hive dialect #15630

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
lirui-apache wants to merge 4 commits into from
Closed

[FLINK-22119][hive][doc] Update document for hive dialect #15630

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
af2a0e7
[FLINK-22119][hive][doc] Update document for hive dialect
lirui-apache Apr 15, 2021
15ca3eb
update doc for dependencies
lirui-apache Apr 16, 2021
ada3dea
address comments
lirui-apache Apr 22, 2021
ab4a972
address comments and sync to zh pages
lirui-apache Apr 23, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
87 changes: 68 additions & 19 deletions docs/content/docs/connectors/table/hive/hive_dialect.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -300,8 +300,6 @@ CREATE VIEW [IF NOT EXISTS] view_name [(column_name, ...) ]

#### Alter

**NOTE**: Altering view only works in Table API, but not supported via SQL client.

##### Rename

```sql
Expand Down Expand Up @@ -346,33 +344,84 @@ CREATE FUNCTION function_name AS class_name;
DROP FUNCTION [IF EXISTS] function_name;
```

## DML
## DML & DQL _`Beta`_

### INSERT
Hive dialect supports a commonly-used subset of Hive's [DML](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML)
and [DQL](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Select). The following lists some examples of
HiveQL supported by the Hive dialect.

```sql
INSERT (INTO|OVERWRITE) [TABLE] table_name [PARTITION partition_spec] SELECT ...;
```
- [SORT/CLUSTER/DISTRIBUTE BY](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+SortBy)
- [Group By](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+GroupBy)
- [Join](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Joins)
- [Union](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Union)
- [LATERAL VIEW](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+LateralView)
- [Window Functions](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+WindowingAndAnalytics)
- [SubQueries](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+SubQueries)
- [CTE](https://cwiki.apache.org/confluence/display/Hive/Common+Table+Expression)
- [INSERT INTO dest schema](https://issues.apache.org/jira/browse/HIVE-9481)
- [Implicit type conversions](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Types#LanguageManualTypes-AllowedImplicitConversions)
Copy link
Member

@wuchong wuchong Apr 22, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are all the supported syntax are listed above? e.g. Select, Join, Union, Subqueries, Lateral views, Over, CTE.

https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Select#LanguageManualSelect-MoreSelectSyntax

Copy link
Contributor Author

@lirui-apache lirui-apache Apr 22, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. I have added these since they're widely used. But it's difficult to make an exhaustive list of supported features.


In order to have better syntax and semantic compatibility, it's highly recommended to use [HiveModule]({{< ref "docs/connectors/table/hive/hive_functions" >}}#use-hive-built-in-functions-via-hivemodule)
and place it first in the module list, so that Hive built-in functions can be picked up during function resolution.

Hive dialect no longer supports [Flink SQL queries]({{< ref "docs/dev/table/sql/queries" >}}). Please switch to `default`
dialect if you'd like to write in Flink syntax.

The `partition_spec`, if present, can be either a full spec or partial spec. If the `partition_spec` is a partial
spec, the dynamic partition column names can be omitted.
Following is an example of using hive dialect to run some queries.

## DQL
```bash
Flink SQL> load module hive;
[INFO] Execute statement succeed.

Flink SQL> use modules hive,core;
[INFO] Execute statement succeed.

At the moment, Hive dialect supports the same syntax as Flink SQL for DQLs. Refer to
[Flink SQL queries]({{< ref "docs/dev/table/sql/queries" >}}) for more details. And it's recommended to switch to
`default` dialect to execute DQLs.
Flink SQL> set table.sql-dialect=hive;
[INFO] Session property has been set.

Flink SQL> select explode(array(1,2,3)); -- call hive udtf
+-----+
| col |
+-----+
| 1 |
| 2 |
| 3 |
+-----+
3 rows in set

Flink SQL> create table tbl (key int,value string);
Copy link
Member

@wuchong wuchong Apr 23, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to create and use a hive catalog before this?

[INFO] Execute statement succeed.

Flink SQL> insert overwrite table tbl values (5,'e'),(1,'a'),(1,'a'),(3,'c'),(2,'b'),(3,'c'),(3,'c'),(4,'d');
[INFO] Submitting SQL update statement to the cluster...
[INFO] SQL update statement has been successfully submitted to the cluster:

Flink SQL> select * from tbl cluster by key; -- run cluster by
2021-04-22 16:13:57,005 INFO org.apache.hadoop.mapred.FileInputFormat [] - Total input paths to process : 1
+-----+-------+
| key | value |
+-----+-------+
| 1 | a |
| 1 | a |
| 5 | e |
| 2 | b |
| 3 | c |
| 3 | c |
| 3 | c |
| 4 | d |
+-----+-------+
8 rows in set
```

## Notice

The following are some precautions for using the Hive dialect.

- Hive dialect should only be used to manipulate Hive tables, not generic tables. And Hive dialect should be used together
with a [HiveCatalog]({{< ref "docs/connectors/table/hive/hive_catalog" >}}).
- Hive dialect should only be used to process Hive meta objects, and requires the current catalog to be a
[HiveCatalog]({{< ref "docs/connectors/table/hive/hive_catalog" >}}).
- Hive dialect only supports 2-part identifiers, so you can't specify catalog for an identifier.
- While all Hive versions support the same syntax, whether a specific feature is available still depends on the
[Hive version]({{< ref "docs/connectors/table/hive/overview" >}}#supported-hive-versions) you use. For example, updating database
location is only supported in Hive-2.4.0 or later.
- Hive and Calcite have different sets of reserved keywords. For example, `default` is a reserved keyword in Calcite and
a non-reserved keyword in Hive. Even with Hive dialect, you have to quote such keywords with backtick ( ` ) in order to
use them as identifiers.
- Due to expanded query incompatibility, views created in Flink cannot be queried in Hive.
- Use [HiveModule]({{< ref "docs/connectors/table/hive/hive_functions" >}}#use-hive-built-in-functions-via-hivemodule)
to run DML and DQL.
24 changes: 24 additions & 0 deletions docs/content/docs/connectors/table/hive/overview.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,9 @@ Please find the required dependencies for different Hive major versions below.
// Hive dependencies
hive-exec-2.3.4.jar

// add antlr-runtime if you need to use hive dialect
antlr-runtime-3.5.2.jar

```
{{< /tab >}}
{{< tab "Hive 1.0.0" >}}
Expand All @@ -150,6 +153,9 @@ Please find the required dependencies for different Hive major versions below.
orc-core-1.4.3-nohive.jar
aircompressor-0.8.jar // transitive dependency of orc-core

// add antlr-runtime if you need to use hive dialect
antlr-runtime-3.5.2.jar

```
{{< /tab >}}
{{< tab "Hive 1.1.0" >}}
Expand All @@ -169,6 +175,9 @@ Please find the required dependencies for different Hive major versions below.
orc-core-1.4.3-nohive.jar
aircompressor-0.8.jar // transitive dependency of orc-core

// add antlr-runtime if you need to use hive dialect
antlr-runtime-3.5.2.jar

```
{{< /tab >}}
{{< tab "Hive 1.2.1" >}}
Expand All @@ -188,6 +197,9 @@ Please find the required dependencies for different Hive major versions below.
orc-core-1.4.3-nohive.jar
aircompressor-0.8.jar // transitive dependency of orc-core

// add antlr-runtime if you need to use hive dialect
antlr-runtime-3.5.2.jar

```
{{< /tab >}}
{{< tab "Hive 2.0.0" >}}
Expand All @@ -201,6 +213,9 @@ Please find the required dependencies for different Hive major versions below.
// Hive dependencies
hive-exec-2.0.0.jar

// add antlr-runtime if you need to use hive dialect
antlr-runtime-3.5.2.jar

```
{{< /tab >}}
{{< tab "Hive 2.1.0" >}}
Expand All @@ -214,6 +229,9 @@ Please find the required dependencies for different Hive major versions below.
// Hive dependencies
hive-exec-2.1.0.jar

// add antlr-runtime if you need to use hive dialect
antlr-runtime-3.5.2.jar

```
{{< /tab >}}
{{< tab "Hive 2.2.0" >}}
Expand All @@ -231,6 +249,9 @@ Please find the required dependencies for different Hive major versions below.
orc-core-1.4.3.jar
aircompressor-0.8.jar // transitive dependency of orc-core

// add antlr-runtime if you need to use hive dialect
antlr-runtime-3.5.2.jar

```
{{< /tab >}}
{{< tab "Hive 3.1.0" >}}
Expand All @@ -245,6 +266,9 @@ Please find the required dependencies for different Hive major versions below.
hive-exec-3.1.0.jar
libfb303-0.9.3.jar // libfb303 is not packed into hive-exec in some versions, need to add it separately

// add antlr-runtime if you need to use hive dialect
antlr-runtime-3.5.2.jar

```
{{< /tab >}}
{{< /tabs >}}
Expand Down