-
Notifications
You must be signed in to change notification settings - Fork 29k
[SPARK-15804][SQL]Include metadata in the toStructType #13555
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 28 commits
3b44c59
18b4a31
4f4d1c8
f5f0cbe
d8b2edb
196b6c6
f37a01e
bb5b01f
bde5820
5f7cd96
893a49a
4bbe1fd
b2dd795
8c3e5da
a0eaa40
d03c940
d728d5e
ea104dd
6ab1215
0c56653
d7a1874
85d3500
c056f91
0b8189d
c2ea31d
a2d3056
1d451d2
200a923
a47dad4
229ca27
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -625,6 +625,21 @@ class ParquetQuerySuite extends QueryTest with ParquetTest with SharedSQLContext | |
| } | ||
| } | ||
| } | ||
|
|
||
| test("SPARK-15804: write out the metadata to parquet file") { | ||
| val df = Seq((1, "abc"), (2, "hello")).toDF("a", "b") | ||
| val md = new MetadataBuilder().putString("key", "value").build() | ||
| val dfWithmeta = df.select('a, 'b.as("b", md)) | ||
|
|
||
| withTempPath { dir => | ||
| val path = s"${dir.getCanonicalPath}/data" | ||
| dfWithmeta.write.parquet(path) | ||
|
|
||
| readParquetFile(path) { df => | ||
| assert(df.schema.json.contains("\"key\":\"value\"")) | ||
|
||
| } | ||
| } | ||
| } | ||
| } | ||
|
|
||
| object TestingUDT { | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think
dir.getCanonicalPathis good here.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok, I will make change