Add tests for vectorized UDF.

[ueshin]

Added tests from apache#19147 and adjusted.

Currently there are 1 failure and 7 errors with these tests.

• failure
  • test_vectorized_udf_invalid_length
• error
  • test_vectorized_udf_null_boolean
  • test_vectorized_udf_null_byte
  • test_vectorized_udf_null_short
  • test_vectorized_udf_null_int
  • test_vectorized_udf_null_long
  • test_vectorized_udf_null_string
  • test_vectorized_udf_zero_parameter
  • test_vectorized_udf_datatype_string

[ueshin]

As for null-related errors, the reason is Pandas changes dtypes if integral type columns contain None. In some cases Arrow throws an exception saying ArrowNotImplementedError, though.
I found workaround like https://github.com/apache/spark/pull/19147/files#diff-e954728a2630cfd7c824e97405d08aafR599 when I implemented before.

[ueshin]

I assumed that you use **kwargs way for size hint.

[BryanCutler]

I'll give it a try and we can see how it turns out, then dscuss

[BryanCutler]

Thanks @ueshin , this looks good. I'll merge it now and work on the errors.

As for null-related errors, the reason is Pandas changes dtypes if integral type columns contain None. In some cases Arrow throws an exception saying ArrowNotImplementedError, though.

Thanks for the pointer, I'll take a look at what's not implemented and see if I can help push that along.

[BryanCutler]

Thanks, merged now. I still need to fix the null-related errors with your suggestion. Would you mind if I used the toPandasSchema you wrote here https://github.com/apache/spark/pull/19147/files#diff-c1cf83efe5a5b1f2f1f770589988e997R1608?

[ueshin]

@BryanCutler Sure, go ahead and use it. Thanks!