FFI Execution Plans that spawn threads panic

[timsaucer]

Describe the bug

If you have an FFI Execution Plan that spawns it's own tasks, it has no reactor and causes a panic.

To Reproduce

This is most easily demonstrated in datafusion-python. Just create a table provider that spawns a thread. I have examples on my computer but they are non-trivial (and it's probably me working on this issue).

Expected behavior

We should never cause a panic across the ffi boundary.

Additional context

No response

[alamb]

Seems like there are two problems:

1. The panic is going across FFI boundaries
2. The FFI code doesn't have a tokio rutime setup

Maybe we need to have some way to pass in a handle to the tokio runtime 🤔 Handle across the FFI boundaries

[timsaucer]

There is only one point where we have ffi calls that are async, currently. This happens in the record batch stream. So I think what needs to happen is that within the private data for the RBS we also include an Option<Arc<Runtime>> and if it exists, we enter the runtime during the async calls.

We do pass a context. I worry about a possible problem where we could have deadlocks if we have two different runtimes each doing a blocking call. The only place that is immediately jumping out to me as a potential problem spot is RecordBatchReceiverStreamBuilder, but I haven't had time to thoroughly look through this.

I think the immediate next thing to do is to build up a good reproducible test, ideally that I could include as a unit test.

[timsaucer]

Initial testing of passing in an Option<Arc<Runtime>> to the table provider and feeding it down through the execution plan and record batch stream was successful.

[kevinjqliu]

Hi @timsaucer thanks for creating this issue!
I'm looking to expose table provider in iceberg-rust to python using FFI_TableProvider. I think I ran into the issue you described above.

This is the current WIP PR https://github.com/kevinjqliu/iceberg-rust/pull/2/files
and here's the error log with backtrace after running hatch run dev:test https://gist.github.com/kevinjqliu/0ab4597427d196c906f279242294d051

I've been looking at reasons why I'm see the error message: there is no reactor running, must be called from the context of a Tokio 1.x runtime. And this issue makes the most sense.

Would love to get your thoughts on this and see if its the same issue

[kevinjqliu]

I tried to incorporate #13937 in kevinjqliu/iceberg-rust#3 and ran into a different error https://gist.github.com/kevinjqliu/cba636cfd4a2a0c8acbba060768e33c9

tests/test_table_provider.py::test_iceberg_table_provider Fatal Python error: Segmentation fault

I'm not sure where the error occurs, but i think integrating the iceberg table provider will be a good test for this issue.

This is my first time working with rust/pyo3/datafusion so there might be something obvious I'm not seeing.

[timsaucer]

I can definitely help and I have a PR to fix the spawn issue. I will be back to work on this next week if you can give me a few dats

[kevinjqliu]

Thank you! Feel free to ping me if you need any help setting up the env for iceberg-rust.

[timsaucer]

Hi Kevin, I'm back to work this week but have a bit of a backlog right now to get through. For the segmentation error above it is probably because you would need to build datafusion-python with that same fix to the FFI API - it is a breaking change.

I have a dev setup on my machine where I should be able to test this. We can potentially get you a work around depending on where you are spawning that task from. The ideal solution is to get my PR here merged in, upgrade datafusion python, and then use that. It might be a month or so until the next datafusion version comes out though.

[kevinjqliu]

For the segmentation error above it is probably because you would need to build datafusion-python with that same fix to the FFI API - it is a breaking change.

Thats exactly what I was missing! I was using datafusion>=43.1.0 in the pytest env .

The ideal solution is to get my PR here merged in, upgrade datafusion python, and then use that.

Looking forward to it! Please let me know if there's anything i can help out with.

Also do you think its a good idea to include Delta and Iceberg TableProviders in the ffitest modules of your PR?