fix: spark array return type mismatch when inner data type is LargeList

[jizezhang]

Which issue does this PR close?

• This PR came up as part of Deduplicate Spark function code with native/default datafusion function code #17964.

Rationale for this change

This PR is intended to fix return type mismatch of spark array when inner data type is LargeList, e.g.

query error
SELECT array(arrow_cast(array(1), 'LargeList(Int64)'))
----
DataFusion error: Internal error: Function 'array' returned value of type 'LargeList(Field { name: "element", data_type: LargeList(Field { data_type: Int64, nullable: true }), nullable: true })' while the following type was promised at planning time and expected: 'List(Field { name: "element", data_type: LargeList(Field { data_type: Int64, nullable: true }), nullable: true })'.
This issue was likely caused by a bug in DataFusion's code. Please help us to resolve this by filing a bug report in our issue tracker: https://github.com/apache/datafusion/issues

What changes are included in this PR?

• Return List regardless of whether inner data type is LargeList or not. This aligns with the behavior of datafusion make_array function.
• Remove return_field_from_args as return_type is already defined and is invoked internally.

Are these changes tested?

Yes

Are there any user-facing changes?

No.

[Jefffrey]

I'm actually thinking maybe we keep this and just fold return_type logic into here; specifically because it returns false for the nullability whereas default implementation for return_field_from_args returns true for nullability.

• We'd still need to implement return_type to satisfy trait, but we can have it return an internal err, see:

datafusion/datafusion/functions-nested/src/map_values.rs

Lines 95 to 112 in b52a81d

	fn return_type(&self, _arg_types: &[DataType]) -> Result<DataType> {
	internal_err!("return_field_from_args should be used instead")
	}
	fn return_field_from_args(
	&self,
	args: datafusion_expr::ReturnFieldArgs,
	) -> Result<FieldRef> {
	let [map_type] = take_function_args(self.name(), args.arg_fields)?;
	Ok(Field::new(
	self.name(),
	DataType::List(get_map_values_field_as_list_field(map_type.data_type())?),
	// Nullable if the map is nullable
	args.arg_fields.iter().any(|x| x.is_nullable()),
	)
	.into())
	}

[jizezhang]

Ah thanks for pointing it out, will raise a revision.

For my learning, make_array implements return_type but not return_field_from_args,

datafusion/datafusion/functions-nested/src/make_array.rs

Line 105 in b52a81d

fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {

which means it uses default implementation of return_field_from_args that has nullable set to true. How would nullability on the return field affect behavior of sql queries? Thanks.

[Jefffrey]

I'm not familiar with the degree to which DataFusion uses the nullability information of a field, so I can't say what the impact would be. But I assume it's better to be more informative where possible, e.g. we know make array function can't ever return a null so it's good if we can preserve that information, especially since the Spark version already had it.

[Jefffrey]

Thanks @jizezhang