[SPARK-36660][SQL] Add cot as Scala and Python functions #33906

yutoacts · 2021-09-03T06:01:52Z

What changes were proposed in this pull request?

Add cotangent support by Dataframe operations (e.g. df.select(cot($"col"))).

Why are the changes needed?

Cotangent has been supported by Spark SQL since 2.3.0 but it cannot be called by Dataframe operations.

Does this PR introduce any user-facing change?

Yes, users can now call the cotangent function by Dataframe operations.

How was this patch tested?

manual test.

yutoacts · 2021-09-03T06:03:38Z

cc @wangyum

wangyum · 2021-09-03T06:36:45Z

cc @HyukjinKwon

HyukjinKwon · 2021-09-03T06:58:33Z

sql/core/src/main/scala/org/apache/spark/sql/functions.scala

+   * @group math_funcs
+   * @since 3.2.0
+   */
+  def cot(e: Column): Column = withExpr { Cot(e.expr) }


Let's only add this one (Column only).

Thanks for the review. I saw other trig functions accept both Column and String(columnName). Should we not use columnName for the parameter?

Yeah, read the guides on the top of f functions.scala

oh, thank you I just removed it.

python/pyspark/sql/functions.py

sql/core/src/main/scala/org/apache/spark/sql/functions.scala

HyukjinKwon · 2021-09-03T07:00:32Z

ok to test

HyukjinKwon · 2021-09-03T07:00:47Z

I am okay with this.
cc @zero323, @WeichenXu123 and @srowen FYI

SparkQA · 2021-09-03T08:07:43Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47469/

Co-authored-by: Hyukjin Kwon <[email protected]>

yutoacts · 2021-09-03T08:31:18Z

BTW noticed that cosecant and secant are missing. Though they are just the reciprocals of sin and cos, I feel a bit odd that only cot is supported.

SparkQA · 2021-09-03T08:51:14Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47469/

SparkQA · 2021-09-03T09:07:45Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47473/

SparkQA · 2021-09-03T09:17:19Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47473/

zero323 · 2021-09-03T10:04:10Z

The arguments here are the same as for SPARK-33061 (#29938). so if we've added those, this one should be fine as well.

SparkQA · 2021-09-03T11:49:44Z

Test build #142969 has finished for PR 33906 at commit ee170d6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-09-03T13:10:28Z

Test build #142972 has finished for PR 33906 at commit c142527.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2021-09-05T08:51:25Z

sql/core/src/main/scala/org/apache/spark/sql/functions.scala

+   * @group math_funcs
+   * @since 3.2.0
+   */
+  def cot(columnName: String): Column = cot(Column(columnName))


Mind removing this one? Looks fine otherwise

Thanks for the reminder. I just removed it.

SparkQA · 2021-09-06T03:07:15Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47508/

SparkQA · 2021-09-06T04:03:31Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47508/

SparkQA · 2021-09-06T04:26:56Z

Test build #143006 has finished for PR 33906 at commit a4ec1e1.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2021-09-06T04:34:25Z

Merged to master.

yutoacts · 2021-09-07T07:13:10Z

I mentioned before but how about implementing SEC/CSC? I believe they can be used as much as COT.

HyukjinKwon · 2021-09-07T07:14:26Z

I am fine with them.

HyukjinKwon · 2021-09-07T07:16:40Z

python/pyspark/sql/functions.py

    return _invoke_function_over_column("cosh", col)


+def cot(col):


actually you should also add them in functions.R too. ref: aeb45df

Sorry, I forgot about that. I'm pushing the follow-up PR soon.

this is fine. usually SparkR one is done separately in a separate JIRA.

yutoacts · 2021-09-07T07:34:03Z

Thank you. I'll post the new ticket and start working on them.

### What changes were proposed in this pull request? Add cotangent as an R function. ### Why are the changes needed? My previous PR (#33906) missed R support. ### Does this PR introduce _any_ user-facing change? Yes, users can now call the cotangent function as an R function. ### How was this patch tested? unit tests. Closes #33925 from yutoacts/SPARK-36660. Lead-authored-by: Yuto Akutsu <[email protected]> Co-authored-by: Yuto Akutsu <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>

sarutak · 2021-09-14T14:58:25Z

We need to change python/docs/source/reference/pyspark.sql.rst for cot right?

HyukjinKwon · 2021-09-14T23:21:08Z

Yup

yutoacts · 2021-09-15T03:05:55Z

Thank you, I just added to it.

### What changes were proposed in this pull request? Added cot to pyspark.sql.rst (follow-up) ### Why are the changes needed? [My previous PR](#33906) was missing it. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? manual check Closes #34002 from yutoacts/SPARK-36660. Authored-by: Yuto Akutsu <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>

### What changes were proposed in this pull request? Add new built-in SQL functions: secant and cosecant, and add them as Scala and Python functions. ### Why are the changes needed? Cotangent has been supported in Spark SQL but Secant and Cosecant are missing though I believe they can be used as much as cot. Related Links: [SPARK-20751](#17999) [SPARK-36660](#33906) ### Does this PR introduce _any_ user-facing change? Yes, users can now use these functions. ### How was this patch tested? Unit tests Closes #33988 from yutoacts/SPARK-36683. Authored-by: Yuto Akutsu <[email protected]> Signed-off-by: Kousuke Saruta <[email protected]>

### What changes were proposed in this pull request? Add new built-in SQL functions: secant and cosecant, and add them as Scala and Python functions. ### Why are the changes needed? Cotangent has been supported in Spark SQL but Secant and Cosecant are missing though I believe they can be used as much as cot. Related Links: [SPARK-20751](apache/spark#17999) [SPARK-36660](apache/spark#33906) ### Does this PR introduce _any_ user-facing change? Yes, users can now use these functions. ### How was this patch tested? Unit tests Closes #33988 from yutoacts/SPARK-36683. Authored-by: Yuto Akutsu <[email protected]> Signed-off-by: Kousuke Saruta <[email protected]>

Yuto Akutsu added 2 commits September 3, 2021 13:40

add cotangent support on DataFrame

c7b7e1d

add cotangent support on PySpark

ee170d6

github-actions bot added CORE PYTHON SQL labels Sep 3, 2021

HyukjinKwon reviewed Sep 3, 2021

View reviewed changes

python/pyspark/sql/functions.py Outdated Show resolved Hide resolved

HyukjinKwon reviewed Sep 3, 2021

View reviewed changes

python/pyspark/sql/functions.py Outdated Show resolved Hide resolved

HyukjinKwon reviewed Sep 3, 2021

View reviewed changes

python/pyspark/sql/functions.py Outdated Show resolved Hide resolved

HyukjinKwon reviewed Sep 3, 2021

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/functions.scala Outdated Show resolved Hide resolved

HyukjinKwon changed the title ~~[SPARK-36660][SQL] Support COT by Dataframe operations~~ [SPARK-36660][SQL] Add cot as Scala and Python functions Sep 3, 2021

Apply suggestions from code review

c142527

Co-authored-by: Hyukjin Kwon <[email protected]>

srowen approved these changes Sep 3, 2021

View reviewed changes

HyukjinKwon reviewed Sep 5, 2021

View reviewed changes

removed cot function with string parameter

a4ec1e1

HyukjinKwon closed this in db95960 Sep 6, 2021

HyukjinKwon reviewed Sep 7, 2021

View reviewed changes

yutoacts mentioned this pull request Sep 7, 2021

[SPARK-36688][R] Add cot as an R function #33925

Closed

yutoacts mentioned this pull request Sep 14, 2021

[SPARK-36683][SQL] Add new built-in SQL functions: SEC and CSC #33988

Closed

yutoacts mentioned this pull request Sep 15, 2021

[SPARK-36660][SQL][FOLLOW-UP] Add cot to pyspark.sql.rst #34002

Closed

		return _invoke_function_over_column("cosh", col)


		def cot(col):

[SPARK-36660][SQL] Add cot as Scala and Python functions #33906

[SPARK-36660][SQL] Add cot as Scala and Python functions #33906

Uh oh!

Conversation

yutoacts commented Sep 3, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

yutoacts commented Sep 3, 2021

Uh oh!

wangyum commented Sep 3, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yutoacts Sep 3, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

HyukjinKwon commented Sep 3, 2021

Uh oh!

HyukjinKwon commented Sep 3, 2021

Uh oh!

SparkQA commented Sep 3, 2021

Uh oh!

yutoacts commented Sep 3, 2021

Uh oh!

SparkQA commented Sep 3, 2021

Uh oh!

SparkQA commented Sep 3, 2021

Uh oh!

SparkQA commented Sep 3, 2021

Uh oh!

zero323 commented Sep 3, 2021

Uh oh!

SparkQA commented Sep 3, 2021

Uh oh!

SparkQA commented Sep 3, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Sep 6, 2021

Uh oh!

SparkQA commented Sep 6, 2021

Uh oh!

SparkQA commented Sep 6, 2021

Uh oh!

HyukjinKwon commented Sep 6, 2021

Uh oh!

yutoacts commented Sep 7, 2021

Uh oh!

HyukjinKwon commented Sep 7, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yutoacts commented Sep 7, 2021

Uh oh!

sarutak commented Sep 14, 2021

Uh oh!

HyukjinKwon commented Sep 14, 2021

Uh oh!

yutoacts commented Sep 15, 2021

yutoacts commented Sep 3, 2021 •

edited

Loading

yutoacts Sep 3, 2021 •

edited

Loading