Skip to content

[SPARK-45070][SQL][DOCS] Describe the binary and datetime formats of to_char/to_varchar #42801

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
MaxGekk wants to merge 6 commits into from
Closed

[SPARK-45070][SQL][DOCS] Describe the binary and datetime formats of to_char/to_varchar #42801

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
c364470
Update Scala/Java APIs
MaxGekk Sep 4, 2023
3385bc8
Update pyspark/sql/functions.py
MaxGekk Sep 4, 2023
992499a
Trigger build
MaxGekk Sep 4, 2023
9d2b782
Address Wenchen's comments
MaxGekk Sep 4, 2023
92a6cac
Reformat functions.scala
MaxGekk Sep 5, 2023
295a50c
Fix python formatting
MaxGekk Sep 5, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/functions.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4263,6 +4263,7 @@ object functions {
*/
def to_binary(e: Column): Column = Column.fn("to_binary", e)

// scalastyle:off line.size.limit
/**
* Convert `e` to a string based on the `format`. Throws an exception if the conversion fails.
*
Expand All @@ -4284,12 +4285,22 @@ object functions {
* prints '+' for positive values but 'MI' prints a space.</li> <li>'PR': Only allowed at the
* end of the format string; specifies that the result string will be wrapped by angle
* brackets if the input value is negative.</li> </ul>
* If `e` is a datetime, `format` shall be a valid datetime pattern, see
* <a href="https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html">Datetime Patterns</a>.
* If `e` is a binary, it is converted to a string in one of the formats:
* <ul>
* <li>'base64': a base 64 string.</li>
* <li>'hex': a string in the hexadecimal format.</li>
* <li>'utf-8': the input binary is decoded to UTF-8 string.</li>
* </ul>
Copy link
Member

@dongjoon-hyun dongjoon-hyun Sep 4, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To @MaxGekk , it seems our Linter CI doesn't allow this kind of human-friendly formatting in the comment.

The scalafmt check failed on connector/connect at following occurrences:

Requires formatting: functions.scala

Before submitting your change, please make sure to format your code using the following command:
./build/mvn -Pscala-2.12 scalafmt:format -Dscalafmt.skip=false -Dscalafmt.validateOnly=false -Dscalafmt.changedOnly=false -pl connector/connect/common -pl connector/connect/server -pl connector/connect/client/jvm
Error: Process completed with exit code 1.

The recommended style is something like the following. Please fix the style by run the above command line.

-   *   brackets if the input value is negative.</li> </ul>
-   *   If `e` is a datetime, `format` shall be a valid datetime pattern, see
-   *   <a href="https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html">Datetime Patterns</a>.
-   *   If `e` is a binary, it is converted to a string in one of the formats:
-   *   <ul>
-   *     <li>'base64': a base 64 string.</li>
-   *     <li>'hex': a string in the hexadecimal format.</li>
-   *     <li>'utf-8': the input binary is decoded to UTF-8 string.</li>
-   *   </ul>
+   *   brackets if the input value is negative.</li> </ul> If `e` is a datetime, `format` shall be
+   *   a valid datetime pattern, see <a
+   *   href="https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html">Datetime
+   *   Patterns</a>. If `e` is a binary, it is converted to a string in one of the formats: <ul>
+   *   <li>'base64': a base 64 string.</li> <li>'hex': a string in the hexadecimal format.</li>
+   *   <li>'utf-8': the input binary is decoded to UTF-8 string.</li> </ul>

Copy link
Member Author

@MaxGekk MaxGekk Sep 5, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, @dongjoon-hyun

*
* @group string_funcs
* @since 3.5.0
*/
// scalastyle:on line.size.limit
def to_char(e: Column, format: Column): Column = Column.fn("to_char", e, format)

// scalastyle:off line.size.limit
/**
* Convert `e` to a string based on the `format`. Throws an exception if the conversion fails.
*
Expand All @@ -4311,10 +4322,19 @@ object functions {
* prints '+' for positive values but 'MI' prints a space.</li> <li>'PR': Only allowed at the
* end of the format string; specifies that the result string will be wrapped by angle
* brackets if the input value is negative.</li> </ul>
* If `e` is a datetime, `format` shall be a valid datetime pattern, see
* <a href="https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html">Datetime Patterns</a>.
* If `e` is a binary, it is converted to a string in one of the formats:
* <ul>
* <li>'base64': a base 64 string.</li>
* <li>'hex': a string in the hexadecimal format.</li>
* <li>'utf-8': the input binary is decoded to UTF-8 string.</li>
* </ul>
*
* @group string_funcs
* @since 3.5.0
*/
// scalastyle:on line.size.limit
def to_varchar(e: Column, format: Column): Column = Column.fn("to_varchar", e, format)

/**
Expand Down
12 changes: 12 additions & 0 deletions python/pyspark/sql/functions.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10423,6 +10423,12 @@ def to_char(col: "ColumnOrName", format: "ColumnOrName") -> Column:
values but 'MI' prints a space.
'PR': Only allowed at the end of the format string; specifies that the result string
will be wrapped by angle brackets if the input value is negative.
If `col` is a datetime, `format` shall be a valid datetime pattern, see
<a href="https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html">Patterns</a>.
If `col` is a binary, it is converted to a string in one of the formats:
'base64': a base 64 string.
'hex': a string in the hexadecimal format.
'utf-8': the input binary is decoded to UTF-8 string.

.. versionadded:: 3.5.0

Expand Down Expand Up @@ -10463,6 +10469,12 @@ def to_varchar(col: "ColumnOrName", format: "ColumnOrName") -> Column:
values but 'MI' prints a space.
'PR': Only allowed at the end of the format string; specifies that the result string
will be wrapped by angle brackets if the input value is negative.
If `col` is a datetime, `format` shall be a valid datetime pattern, see
<a href="https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html">Patterns</a>.
If `col` is a binary, it is converted to a string in one of the formats:
'base64': a base 64 string.
'hex': a string in the hexadecimal format.
'utf-8': the input binary is decoded to UTF-8 string.

.. versionadded:: 3.5.0

Expand Down
18 changes: 18 additions & 0 deletions sql/core/src/main/scala/org/apache/spark/sql/functions.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4399,6 +4399,7 @@ object functions {
new ToBinary(e.expr)
}

// scalastyle:off line.size.limit
/**
* Convert `e` to a string based on the `format`.
* Throws an exception if the conversion fails. The format can consist of the following
Expand All @@ -4420,11 +4421,20 @@ object functions {
* 'PR': Only allowed at the end of the format string; specifies that the result string will be
* wrapped by angle brackets if the input value is negative.
*
* If `e` is a datetime, `format` shall be a valid datetime pattern, see
* <a href="https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html">Datetime Patterns</a>.
* If `e` is a binary, it is converted to a string in one of the formats:
* 'base64': a base 64 string.
* 'hex': a string in the hexadecimal format.
* 'utf-8': the input binary is decoded to UTF-8 string.
*
* @group string_funcs
* @since 3.5.0
*/
// scalastyle:on line.size.limit
def to_char(e: Column, format: Column): Column = call_function("to_char", e, format)

// scalastyle:off line.size.limit
/**
* Convert `e` to a string based on the `format`.
* Throws an exception if the conversion fails. The format can consist of the following
Expand All @@ -4446,9 +4456,17 @@ object functions {
* 'PR': Only allowed at the end of the format string; specifies that the result string will be
* wrapped by angle brackets if the input value is negative.
*
* If `e` is a datetime, `format` shall be a valid datetime pattern, see
* <a href="https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html">Datetime Patterns</a>.
* If `e` is a binary, it is converted to a string in one of the formats:
* 'base64': a base 64 string.
* 'hex': a string in the hexadecimal format.
* 'utf-8': the input binary is decoded to UTF-8 string.
*
* @group string_funcs
* @since 3.5.0
*/
// scalastyle:on line.size.limit
def to_varchar(e: Column, format: Column): Column = call_function("to_varchar", e, format)

/**
Expand Down