Skip to content

[SPARK-13509][SPARK-13507][SQL] Support for writing CSV with a single function call #11389

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
HyukjinKwon wants to merge 9 commits into from
Closed

[SPARK-13509][SPARK-13507][SQL] Support for writing CSV with a single function call #11389

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
a97a0a8
Support for writing CSV with a single function call
HyukjinKwon Feb 26, 2016
f82a2f4
Move the comments above for json.
HyukjinKwon Feb 26, 2016
9fe8fca
Update comments and remove the supprot for `codec` for JSON and TEXT
HyukjinKwon Feb 29, 2016
c703def
Add Python API
HyukjinKwon Feb 29, 2016
8efb0e3
Update upstream
HyukjinKwon Feb 29, 2016
8fbad40
Add a newline at the end of the test file
HyukjinKwon Feb 29, 2016
9ca920b
Update an weired indentation
HyukjinKwon Feb 29, 2016
fea9df8
Make path seq if it is an string
HyukjinKwon Feb 29, 2016
cec8442
Read without options
HyukjinKwon Feb 29, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -464,6 +464,12 @@ final class DataFrameWriter private[sql](df: DataFrame) {
* format("parquet").save(path)
* }}}
*
* You can set the following JSON-specific options for writing JSON files:
Copy link
Member

@srowen srowen Feb 26, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like it's in the wrong place?

* <li>`compression` or `codec` (default `null`): compression codec to use when saving to file.
* This should be the fully qualified name of a class implementing
* [[org.apache.hadoop.io.compress.CompressionCodec]] or one of the known case-insensitive
* shorten names(`bzip2`, `gzip`, `lz4`, and `snappy`). </li>
*
* @since 1.4.0
*/
def parquet(path: String): Unit = format("parquet").save(path)
Expand Down Expand Up @@ -492,10 +498,33 @@ final class DataFrameWriter private[sql](df: DataFrame) {
* df.write().text("/path/to/output")
* }}}
*
* You can set the following options for writing text files:
* <li>`compression` or `codec` (default `null`): compression codec to use when saving to file.
* This should be the fully qualified name of a class implementing
* [[org.apache.hadoop.io.compress.CompressionCodec]] or one of the known case-insensitive
* shorten names(`bzip2`, `gzip`, `lz4`, and `snappy`). </li>
*
* @since 1.6.0
*/
def text(path: String): Unit = format("text").save(path)

/**
* Saves the content of the [[DataFrame]] in CSV format at the specified path.
* This is equivalent to:
* {{{
* format("csv").save(path)
* }}}
*
* You can set the following CSV-specific options for writing CSV files:
* <li>`compression` or `codec` (default `null`): compression codec to use when saving to file.
* This should be the fully qualified name of a class implementing
* [[org.apache.hadoop.io.compress.CompressionCodec]] or one of the known case-insensitive
* shorten names(`bzip2`, `gzip`, `lz4`, and `snappy`). </li>
*
* @since 2.0.0
*/
def csv(path: String): Unit = format("csv").save(path)

///////////////////////////////////////////////////////////////////////////////////////
// Builder pattern config options
///////////////////////////////////////////////////////////////////////////////////////
Expand Down
3 changes: 1 addition & 2 deletions sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -268,9 +268,8 @@ class CSVSuite extends QueryTest with SharedSQLContext with SQLTestUtils {
.load(testFile(carsFile))

cars.coalesce(1).write
.format("csv")
.option("header", "true")
.save(csvDir)
.csv(csvDir)

val carsCopy = sqlContext.read
.format("csv")
Expand Down