2121
2222## DataFusion ` 46.0.0 `
2323
24+ ### Use ` invoke_with_args ` instead of ` invoke() ` and ` invoke_batch() `
2425
25- ### Changes to ` invoke() ` and ` invoke_batch() ` deprecated
26+ DataFusion is moving to a consistent API for invoking ScalarUDFs,
27+ [ ` ScalarUDFImpl::invoke_with_args() ` ] , and deprecating
28+ [ ` ScalarUDFImpl::invoke() ` ] , [ ` ScalarUDFImpl::invoke_batch() ` ] , and [ ` ScalarUDFImpl::invoke_no_args() ` ]
2629
27- We are migrating away from ` ScalarUDFImpl::invoke() ` and
28- ` ScalarUDFImpl::invoke_batch() ` in favor of ` ScalarUDFImpl::invoke_with_args() ` . (TODO get code links)
30+ If you see errors such as the following it means the older APIs are being used
2931
30- If you see errors such as
3132``` text
32- Example
33+ This feature is not implemented: Function concat does not implement invoke but called
3334```
3435
35- You can resolve them by replacing all .invoke() and .invoke_batch()calls with .invoke_with_args().
36- ``` text
37- TODO example
38- ```
36+ To fix this error, change your functions to use
37+ [ ` ScalarUDFImpl::invoke_with_args() ` ] instea, as shown below. See [ PR 14876] for
38+ an example.
39+
40+ Given existing code like this:
41+
42+ ```` rust
43+ # /*
44+ impl ScalarUDFImpl for SparkConcat {
45+ ...
46+ fn invoke_batch(&self, args: &[ColumnarValue], number_rows: usize) -> Result<ColumnarValue> {
47+ if args
48+ .iter()
49+ .any(|arg| matches!(arg.data_type(), DataType::List(_)))
50+ {
51+ ArrayConcat::new().invoke_batch(args, number_rows)
52+ } else {
53+ ConcatFunc::new().invoke_batch(args, number_rows)
54+ }
55+ }
56+ # */
57+ }```
3958
40- Example of changes:
41- - [ PR XXXX] TODO
59+ To
4260
61+ ```rust
62+ # /* comment out so they don't run
63+ impl ScalarUDFImpl for SparkConcat {
64+ ...
65+ fn invoke_with_args(&self, args: ScalarFunctionArgs) -> Result<ColumnarValue> {
66+ if args
67+ .args
68+ .iter()
69+ .any(|arg| matches!(arg.data_type(), DataType::List(_)))
70+ {
71+ ArrayConcat::new().invoke_with_args(args)
72+ } else {
73+ ConcatFunc::new().invoke_with_args(args)
74+ }
75+ }
76+ }
77+ # */
78+ ````
79+
80+ [ `scalarudfimpl::invoke()` ] : https://docs.rs/datafusion/latest/datafusion/logical_expr/trait.ScalarUDFImpl.html#method.invoke
81+ [ `scalarudfimpl::invoke_batch()` ] : https://docs.rs/datafusion/latest/datafusion/logical_expr/trait.ScalarUDFImpl.html#method.invoke_batch
82+ [ `scalarudfimpl::invoke_no_args()` ] : https://docs.rs/datafusion/latest/datafusion/logical_expr/trait.ScalarUDFImpl.html#method.invoke_no_args
83+ [ `scalarudfimpl::invoke_with_args()` ] : https://docs.rs/datafusion/latest/datafusion/logical_expr/trait.ScalarUDFImpl.html#method.invoke_with_args
84+ [ pr 14876 ] : https://github.com/apache/datafusion/pull/14876
4385
4486### ` ParquetExec ` , ` AvroExec ` , ` CsvExec ` , ` JsonExec ` deprecated
4587
4688See more information
89+
4790- Change PR [ PR #14224 ] ( https://github.com/apache/datafusion/pull/14224 )
4891- Example of an Upgrade [ PR in delta-rs] ( https://github.com/delta-io/delta-rs/pull/3261 )
4992
50- DataFusion 46 has a major change to how the built in DataSources are organized. The
93+ DataFusion 46 has a major change to how the built in DataSources are organized. The
5194
5295### Cookbook: Changes to ` ParquetExecBuilder `
96+
5397#### Old pattern:
5498
5599When writing optimizer passes, some code treats ParquetExec specially like this:
@@ -62,14 +106,15 @@ When writing optimizer passes, some code treats ParquetExec specially like this:
62106```
63107
64108#### New Pattern
65- With the new DataSource exec, most information is now on ` FileScanConfig ` and ` ParquetSource `
109+
110+ With the new DataSource exec, most information is now on ` FileScanConfig ` and ` ParquetSource `
66111
67112``` rust
68113
69114if let Some (datasource_exec ) = plan . as_any (). downcast_ref :: <DataSourceExec >() {
70115 if let Some (scan_config ) = datasource_exec . source (). as_any (). downcast_ref :: <FileScanConfig >() {
71116 // FileGroups, and other information is on the FileScanConfig
72- // parquet
117+ // parquet
73118 if let Some (parquet_source ) = scan_config . source. as_any (). downcast_ref :: <ParquetSource >()
74119 {
75120 // Information on PruningPredicates and parquet options are here
@@ -81,7 +126,7 @@ if let Some(datasource_exec) = plan.as_any().downcast_ref::<DataSourceExec>() {
81126
82127#### Old pattern:
83128
84- ``` rust
129+ ```` rust
85130 let mut exec_plan_builder = ParquetExecBuilder :: new (
86131 FileScanConfig :: new (self . log_store. object_store_url (), file_schema )
87132 . with_file_groups (
@@ -152,8 +197,7 @@ if let Some(datasource_exec) = plan.as_any().downcast_ref::<DataSourceExec>() {
152197// Build the actual scan like this
153198parquet_scan : file_scan_config . build (),
154199
155- ```
156-
200+ ````
157201
158202### ` datafusion-cli ` no longer automatically unescapes strings
159203
0 commit comments