Skip to content

Commit 9c658c7

Browse files
committed
Cleanup ScalarUDFImpl docs
1 parent 074f5ec commit 9c658c7

File tree

1 file changed

+61
-17
lines changed

1 file changed

+61
-17
lines changed

docs/source/library-user-guide/upgrading.md

Lines changed: 61 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -21,35 +21,79 @@
2121

2222
## DataFusion `46.0.0`
2323

24+
### Use `invoke_with_args` instead of `invoke()` and `invoke_batch()`
2425

25-
### Changes to `invoke()` and `invoke_batch()` deprecated
26+
DataFusion is moving to a consistent API for invoking ScalarUDFs,
27+
[`ScalarUDFImpl::invoke_with_args()`], and deprecating
28+
[`ScalarUDFImpl::invoke()`], [`ScalarUDFImpl::invoke_batch()`], and [`ScalarUDFImpl::invoke_no_args()`]
2629

27-
We are migrating away from `ScalarUDFImpl::invoke()` and
28-
`ScalarUDFImpl::invoke_batch()` in favor of `ScalarUDFImpl::invoke_with_args()`. (TODO get code links)
30+
If you see errors such as the following it means the older APIs are being used
2931

30-
If you see errors such as
3132
```text
32-
Example
33+
This feature is not implemented: Function concat does not implement invoke but called
3334
```
3435

35-
You can resolve them by replacing all .invoke() and .invoke_batch()calls with .invoke_with_args().
36-
```text
37-
TODO example
38-
```
36+
To fix this error, change your functions to use
37+
[`ScalarUDFImpl::invoke_with_args()`] instea, as shown below. See [PR 14876] for
38+
an example.
39+
40+
Given existing code like this:
41+
42+
````rust
43+
# /*
44+
impl ScalarUDFImpl for SparkConcat {
45+
...
46+
fn invoke_batch(&self, args: &[ColumnarValue], number_rows: usize) -> Result<ColumnarValue> {
47+
if args
48+
.iter()
49+
.any(|arg| matches!(arg.data_type(), DataType::List(_)))
50+
{
51+
ArrayConcat::new().invoke_batch(args, number_rows)
52+
} else {
53+
ConcatFunc::new().invoke_batch(args, number_rows)
54+
}
55+
}
56+
# */
57+
}```
3958

40-
Example of changes:
41-
- [PR XXXX] TODO
59+
To
4260

61+
```rust
62+
# /* comment out so they don't run
63+
impl ScalarUDFImpl for SparkConcat {
64+
...
65+
fn invoke_with_args(&self, args: ScalarFunctionArgs) -> Result<ColumnarValue> {
66+
if args
67+
.args
68+
.iter()
69+
.any(|arg| matches!(arg.data_type(), DataType::List(_)))
70+
{
71+
ArrayConcat::new().invoke_with_args(args)
72+
} else {
73+
ConcatFunc::new().invoke_with_args(args)
74+
}
75+
}
76+
}
77+
# */
78+
````
79+
80+
[`scalarudfimpl::invoke()`]: https://docs.rs/datafusion/latest/datafusion/logical_expr/trait.ScalarUDFImpl.html#method.invoke
81+
[`scalarudfimpl::invoke_batch()`]: https://docs.rs/datafusion/latest/datafusion/logical_expr/trait.ScalarUDFImpl.html#method.invoke_batch
82+
[`scalarudfimpl::invoke_no_args()`]: https://docs.rs/datafusion/latest/datafusion/logical_expr/trait.ScalarUDFImpl.html#method.invoke_no_args
83+
[`scalarudfimpl::invoke_with_args()`]: https://docs.rs/datafusion/latest/datafusion/logical_expr/trait.ScalarUDFImpl.html#method.invoke_with_args
84+
[pr 14876]: https://github.com/apache/datafusion/pull/14876
4385

4486
### `ParquetExec`, `AvroExec`, `CsvExec`, `JsonExec` deprecated
4587

4688
See more information
89+
4790
- Change PR [PR #14224](https://github.com/apache/datafusion/pull/14224)
4891
- Example of an Upgrade [PR in delta-rs](https://github.com/delta-io/delta-rs/pull/3261)
4992

50-
DataFusion 46 has a major change to how the built in DataSources are organized. The
93+
DataFusion 46 has a major change to how the built in DataSources are organized. The
5194

5295
### Cookbook: Changes to `ParquetExecBuilder`
96+
5397
#### Old pattern:
5498

5599
When writing optimizer passes, some code treats ParquetExec specially like this:
@@ -62,14 +106,15 @@ When writing optimizer passes, some code treats ParquetExec specially like this:
62106
```
63107

64108
#### New Pattern
65-
With the new DataSource exec, most information is now on `FileScanConfig` and `ParquetSource`
109+
110+
With the new DataSource exec, most information is now on `FileScanConfig` and `ParquetSource`
66111

67112
```rust
68113

69114
if let Some(datasource_exec) = plan.as_any().downcast_ref::<DataSourceExec>() {
70115
if let Some(scan_config) = datasource_exec.source().as_any().downcast_ref::<FileScanConfig>() {
71116
// FileGroups, and other information is on the FileScanConfig
72-
// parquet
117+
// parquet
73118
if let Some(parquet_source) = scan_config.source.as_any().downcast_ref::<ParquetSource>()
74119
{
75120
// Information on PruningPredicates and parquet options are here
@@ -81,7 +126,7 @@ if let Some(datasource_exec) = plan.as_any().downcast_ref::<DataSourceExec>() {
81126

82127
#### Old pattern:
83128

84-
```rust
129+
````rust
85130
let mut exec_plan_builder = ParquetExecBuilder::new(
86131
FileScanConfig::new(self.log_store.object_store_url(), file_schema)
87132
.with_file_groups(
@@ -152,8 +197,7 @@ if let Some(datasource_exec) = plan.as_any().downcast_ref::<DataSourceExec>() {
152197
// Build the actual scan like this
153198
parquet_scan: file_scan_config.build(),
154199

155-
```
156-
200+
````
157201

158202
### `datafusion-cli` no longer automatically unescapes strings
159203

0 commit comments

Comments
 (0)