Skip to content

Commit ddbeb4d

Browse files
rdbluerdblue
committed
PARQUET-777: Add Parquet CLI.
This adds a new parquet-cli module with an improved command-line tool. The parquet-cli/README.md file has instructions for building and testing locally. Author: Ryan Blue <blue@apache.org> Author: Tom White <tom@cloudera.com> Closes #384 from rdblue/PARQUET-777-add-parquet-cli and squashes the following commits: de49eff [Ryan Blue] PARQUET-777: Move dynamic support classes, add tests. affdfb9 [Ryan Blue] PARQUET-777: Update for review feedback. f953fd4 [Ryan Blue] PARQUET-777: Update README.md with better instructions. aed223d [Tom White] Replace source file headers with Apache header. d718363 [Ryan Blue] PARQUET-777: Add Parquet CLI.
1 parent df9f8d8 commit ddbeb4d

44 files changed

Lines changed: 7909 additions & 0 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎NOTICE‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -54,3 +54,41 @@ its NOTICE file:
5454
This product includes software developed at
5555
The Apache Software Foundation (http://www.apache.org/).
5656

57+
--------------------------------------------------------------------------------
58+
59+
This project includes code from Kite, developed at Cloudera, Inc. with
60+
the following copyright notice:
61+
62+
| Copyright 2013 Cloudera Inc.
63+
|
64+
| Licensed under the Apache License, Version 2.0 (the "License");
65+
| you may not use this file except in compliance with the License.
66+
| You may obtain a copy of the License at
67+
|
68+
| http://www.apache.org/licenses/LICENSE-2.0
69+
|
70+
| Unless required by applicable law or agreed to in writing, software
71+
| distributed under the License is distributed on an "AS IS" BASIS,
72+
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
73+
| See the License for the specific language governing permissions and
74+
| limitations under the License.
75+
76+
--------------------------------------------------------------------------------
77+
78+
This project includes code from Netflix, Inc. with the following copyright
79+
notice:
80+
81+
| Copyright 2016 Netflix, Inc.
82+
|
83+
| Licensed under the Apache License, Version 2.0 (the "License");
84+
| you may not use this file except in compliance with the License.
85+
| You may obtain a copy of the License at
86+
|
87+
| http://www.apache.org/licenses/LICENSE-2.0
88+
|
89+
| Unless required by applicable law or agreed to in writing, software
90+
| distributed under the License is distributed on an "AS IS" BASIS,
91+
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
92+
| See the License for the specific language governing permissions and
93+
| limitations under the License.
94+

‎parquet-cli/README.md‎

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
<!--
2+
- Licensed to the Apache Software Foundation (ASF) under one
3+
- or more contributor license agreements. See the NOTICE file
4+
- distributed with this work for additional information
5+
- regarding copyright ownership. The ASF licenses this file
6+
- to you under the Apache License, Version 2.0 (the
7+
- "License"); you may not use this file except in compliance
8+
- with the License. You may obtain a copy of the License at
9+
-
10+
- http://www.apache.org/licenses/LICENSE-2.0
11+
-
12+
- Unless required by applicable law or agreed to in writing,
13+
- software distributed under the License is distributed on an
14+
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
- KIND, either express or implied. See the License for the
16+
- specific language governing permissions and limitations
17+
- under the License.
18+
-->
19+
20+
## Building
21+
22+
You can build this project using maven:
23+
24+
```
25+
mvn clean install -DskipTests
26+
```
27+
28+
29+
## Running
30+
31+
The build produces a shaded Jar that can be run using the `hadoop` command:
32+
33+
```
34+
hadoop jar parquet-cli-1.9.1-runtime.jar org.apache.parquet.cli.Main
35+
```
36+
37+
For a shorter command-line invocation, add an alias to your shell like this:
38+
39+
```
40+
alias parquet="hadoop jar /path/to/parquet-cli-1.9.1-runtime.jar org.apache.parquet.cli.Main --dollar-zero parquet"
41+
```
42+
43+
### Running without Hadoop
44+
45+
To run from the target directory instead of using the `hadoop` command, first copy the dependencies to a folder:
46+
47+
```
48+
mvn dependency:copy-dependencies
49+
```
50+
51+
Then, run the command-line and add `target/dependencies/*` to the classpath:
52+
53+
```
54+
java -cp 'target/*:target/dependency/*' org.apache.parquet.cli.Main
55+
```
56+
57+
58+
### Help
59+
60+
The `parquet` tool includes help for the included commands:
61+
62+
```
63+
parquet help
64+
```
65+
```
66+
Usage: parquet [options] [command] [command options]
67+
68+
Options:
69+
70+
-v, --verbose, --debug
71+
Print extra debugging information
72+
73+
Commands:
74+
75+
help
76+
Retrieves details on the functions of other commands
77+
meta
78+
Print a Parquet file's metadata
79+
pages
80+
Print page summaries for a Parquet file
81+
dictionary
82+
Print dictionaries for a Parquet column
83+
check-stats
84+
Check Parquet files for corrupt page and column stats (PARQUET-251)
85+
schema
86+
Print the Avro schema for a file
87+
csv-schema
88+
Build a schema from a CSV data sample
89+
convert-csv
90+
Create a file from CSV data
91+
convert
92+
Create a Parquet file from a data file
93+
to-avro
94+
Create an Avro file from a data file
95+
cat
96+
Print the first N records from a file
97+
head
98+
Print the first N records from a file
99+
100+
Examples:
101+
102+
# print information for create
103+
parquet help create
104+
105+
See 'parquet help <command>' for more information on a specific command.
106+
```
107+

‎parquet-cli/pom.xml‎

Lines changed: 153 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,153 @@
1+
<!--
2+
~ Licensed to the Apache Software Foundation (ASF) under one
3+
~ or more contributor license agreements. See the NOTICE file
4+
~ distributed with this work for additional information
5+
~ regarding copyright ownership. The ASF licenses this file
6+
~ to you under the Apache License, Version 2.0 (the
7+
~ "License"); you may not use this file except in compliance
8+
~ with the License. You may obtain a copy of the License at
9+
~
10+
~ http://www.apache.org/licenses/LICENSE-2.0
11+
~
12+
~ Unless required by applicable law or agreed to in writing,
13+
~ software distributed under the License is distributed on an
14+
~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
~ KIND, either express or implied. See the License for the
16+
~ specific language governing permissions and limitations
17+
~ under the License.
18+
-->
19+
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
20+
<parent>
21+
<groupId>org.apache.parquet</groupId>
22+
<artifactId>parquet</artifactId>
23+
<relativePath>../pom.xml</relativePath>
24+
<version>1.9.1-SNAPSHOT</version>
25+
</parent>
26+
27+
<modelVersion>4.0.0</modelVersion>
28+
29+
<artifactId>parquet-cli</artifactId>
30+
<packaging>jar</packaging>
31+
32+
<name>Apache Parquet Command-line</name>
33+
<url>https://parquet.apache.org</url>
34+
35+
<dependencies>
36+
<dependency>
37+
<groupId>org.apache.parquet</groupId>
38+
<artifactId>parquet-avro</artifactId>
39+
<version>${project.version}</version>
40+
</dependency>
41+
<dependency>
42+
<groupId>org.apache.avro</groupId>
43+
<artifactId>avro</artifactId>
44+
<version>${avro.version}</version>
45+
</dependency>
46+
<dependency>
47+
<groupId>org.slf4j</groupId>
48+
<artifactId>slf4j-api</artifactId>
49+
<version>${slf4j.version}</version>
50+
</dependency>
51+
<dependency>
52+
<groupId>net.sf.opencsv</groupId>
53+
<artifactId>opencsv</artifactId>
54+
<version>${opencsv.version}</version>
55+
</dependency>
56+
<dependency>
57+
<groupId>com.fasterxml.jackson.core</groupId>
58+
<artifactId>jackson-databind</artifactId>
59+
<version>${jackson2.version}</version>
60+
</dependency>
61+
<dependency>
62+
<groupId>com.beust</groupId>
63+
<artifactId>jcommander</artifactId>
64+
<version>${jcommander.version}</version>
65+
</dependency>
66+
67+
<dependency>
68+
<groupId>org.slf4j</groupId>
69+
<artifactId>slf4j-log4j12</artifactId>
70+
<version>${slf4j.version}</version>
71+
<scope>provided</scope>
72+
</dependency>
73+
<dependency>
74+
<groupId>com.google.guava</groupId>
75+
<artifactId>guava</artifactId>
76+
<version>${guava.version}</version>
77+
<scope>provided</scope>
78+
</dependency>
79+
<dependency>
80+
<groupId>commons-codec</groupId>
81+
<artifactId>commons-codec</artifactId>
82+
<version>${commons-codec.version}</version>
83+
<scope>provided</scope>
84+
</dependency>
85+
<dependency>
86+
<groupId>org.apache.hadoop</groupId>
87+
<artifactId>hadoop-client</artifactId>
88+
<version>${hadoop.version}</version>
89+
<scope>provided</scope>
90+
</dependency>
91+
</dependencies>
92+
93+
<build>
94+
<plugins>
95+
<!-- This module disables semver checks because it is not a public API.
96+
<plugin>
97+
<artifactId>maven-enforcer-plugin</artifactId>
98+
</plugin>
99+
-->
100+
<plugin>
101+
<groupId>org.apache.maven.plugins</groupId>
102+
<artifactId>maven-jar-plugin</artifactId>
103+
</plugin>
104+
<plugin>
105+
<groupId>org.apache.maven.plugins</groupId>
106+
<artifactId>maven-shade-plugin</artifactId>
107+
<executions>
108+
<execution>
109+
<phase>package</phase>
110+
<goals>
111+
<goal>shade</goal>
112+
</goals>
113+
<configuration>
114+
<shadedArtifactAttached>true</shadedArtifactAttached>
115+
<shadedClassifierName>runtime</shadedClassifierName>
116+
<minimizeJar>false</minimizeJar>
117+
<filters>
118+
<filter>
119+
<artifact>org.xerial.snappy:*</artifact>
120+
<excludes>
121+
<exclude>**/LICENSE</exclude>
122+
</excludes>
123+
</filter>
124+
<filter>
125+
<artifact>*:*</artifact>
126+
<excludes>
127+
<exclude>META-INF/LICENSE.txt</exclude>
128+
<exclude>META-INF/NOTICE.txt</exclude>
129+
</excludes>
130+
</filter>
131+
</filters>
132+
<artifactSet>
133+
<includes>
134+
<include>*</include>
135+
</includes>
136+
</artifactSet>
137+
<relocations>
138+
<relocation>
139+
<!-- relocate Avro in the runtime jar to avoid conflicts with
140+
on-cluster Avro versions.
141+
-->
142+
<pattern>org.apache.avro</pattern>
143+
<shadedPattern>${shade.prefix}.org.apache.avro</shadedPattern>
144+
</relocation>
145+
</relocations>
146+
</configuration>
147+
</execution>
148+
</executions>
149+
</plugin>
150+
</plugins>
151+
</build>
152+
153+
</project>

0 commit comments

Comments
 (0)