[C++] compute kernels registry is not reset when dll is unloaded

[alinaliBQ]

Describe the bug, including details regarding any error messages, version, and platform.

Hi community, it seems the compute kernels registry is not cleared when the dll library (that is dynamically linked to arrow_compute.dll) is unloaded. Is this expected behavior? If not, then this might be a big impact.

Please let me know if you have any questions. Here is my attempt at explaining the issue that our team is facing:

• we are working on an Arrow Flight SQL ODBC driver dll (arrow_flight_sql_odbc.dll) library at GH-46734: [C++] Arrow Flight SQL ODBC layer #46099, that uses arrow_compute.dll
• arrow::compute::Initialize() is called once from arrow_flight_sql_odbc.dll every time the driver dll is loaded in the test executable. And each test case loads the driver dll at the beginning and unloads at the end.
• this means arrow::compute::Initialize() gets called every test case
• this causes an issue because when the second test case starts, we receive Already have a function registered with name error and it blocks rest of tests from running.

What the error message looks like:

[Test 1 passes]
[Test 2 starts]
arrow\cpp\src\arrow\compute\kernels\scalar_arithmetic.cc:1448:  Check failed: _s.ok() Operation failed: registry->AddFunction(std::move(absolute_value))
Bad status: Key error: Already have a function registered with name: abs

• this error message makes sense because abs is the first function that gets registered, and Arrow errors out on this first call to AddFunction.

I think that every dll library that calls arrow::compute::Initialize() could potentially run into this issue. This is because the application that uses the dll isn't guaranteed to know Arrow compute, so the application isn't expected to call arrow::compute::Initialize().

• I can confirm this error goes away when I remove the arrow::compute::Initialize() call from the driver dll and into the test environment set up, but the initialization needs to happen inside the driver, as the driver needs to initialize the compute functions in order to work properly.

Thanks in advance to people that are looking into this.

Component(s)

C++

[alinaliBQ]

Hi @raulcd @zanmato1984 @kou the commit history shows you guys have worked on Arrow compute kernel recently in the past, do you have any insights on this?
cc @lidavidm @jduo

[lidavidm]

I'm not sure we support being unloaded in general.

[raulcd]

We do not support unload. I don't have the full context on the odbc driver but the arrow::compute::Initialize is supposed to be called from the user perspective, example when users use acero or other modules we don't automatically initialize the compute functions for them is the user which requires to do so. On tests for those we have a custom main that does this for us automatically.

If you have to register the compute kernels when the flight sql odbc driver is loaded and you must do it on that function I would do it conditionally instead of throwing the error. Check if the function required is registered and only register if not. Raise the error if it fails only on the case?

[alinaliBQ]

I wasn't aware that unloading isn't supported yet. Thanks for letting me know.
Let me look into loading the kernel registry conditionally, the suggested approach makes sense. Thanks

[alinaliBQ]

I am happy to reply that registering function conditionally worked for me. Here is what I used:

void FlightSqlDriver::RegisterComputeKernels() {
  auto registry = arrow::compute::GetFunctionRegistry();

  // strptime is one of the required compute functions
  auto strptime_func = registry->GetFunction("strptime");
  if (!strptime_func.ok()) {
    // Register Kernel functions to library
    ThrowIfNotOK(arrow::compute::Initialize());
  }
}