Latest Enzyme version broke DI tests with new runtime activity error

[gdalle]

Enzyme v0.13.109 broke DI's second-order tests, and it probably has something to do with BatchDuplicated.

Here's a DI-based MWE while I extract its non-DI counterpart:

using DifferentiationInterface
import Enzyme

arr_to_num(x::AbstractArray) = sum(vec(x .^ 4) .* transpose(vec(x .^ 6)))

function f(x, c)
    c[1] = arr_to_num(x)
    return c[1]
end

x = [3.0, 5.0];
dx1, dx2 = [1.0, 0.0], [0.0, 1.0];
c = [0.0];

backend = AutoEnzyme()

# works on all Enzyme versions
hvp(f, backend, x, (dx1,), Cache(c))

# works on Enzyme v0.13.108, fails on Enzyme v0.13.109
hvp(f, backend, x, (dx1, dx2), Cache(c))

Stack trace:

julia> hvp(f, backend, x, (dx1, dx2), Cache(c))
ERROR: EnzymeRuntimeActivityError: Detected potential need for runtime activity.

Constant memory is stored (or returned) to a differentiable variable and correctness cannot be guaranteed with static activity analysis.
This might be due to the use of a constant variable as temporary storage for active memory (https://enzyme.mit.edu/julia/stable/faq/#Runtime-Activity).
If Enzyme should be able to prove this use non-differentable, open an issue!

To work around this issue, either:
   a) rewrite this variable to not be conditionally active (fastest performance, slower to setup), or
   b) set the Enzyme mode to turn on runtime activity (e.g. autodiff(set_runtime_activity(Reverse), ...) ). This will maintain correctness, but may slightly reduce performance.


Mismatched activity for:   %.fca.0.5.insert = insertvalue { { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, i64, double*, {} addrspace(10)**, double*, double*, double* }, double } %.fca.0.4.insert, {} addrspace(10)** %.sroa.13.0, 0, 5, !dbg !307 const val:   %.sroa.13.0 = phi {} addrspace(10)** [ %37, %L113.thread ], [ undef, %L207.loopexit ]
Type tree: {[0]:Pointer, [8]:Pointer, [8,0]:Integer, [8,1]:Integer, [8,2]:Integer, [8,3]:Integer, [8,4]:Integer, [8,5]:Integer, [8,6]:Integer, [8,7]:Integer, [8,8]:Pointer, [8,8,-1]:Float@double, [16]:Pointer, [24]:Integer, [25]:Integer, [26]:Integer, [27]:Integer, [28]:Integer, [29]:Integer, [30]:Integer, [31]:Integer, [32]:Pointer, [32,0]:Float@double, [40]:Pointer, [48]:Anything, [49]:Anything, [50]:Anything, [51]:Anything, [52]:Anything, [53]:Anything, [54]:Anything, [55]:Anything, [56]:Anything, [57]:Anything, [58]:Anything, [59]:Anything, [60]:Anything, [61]:Anything, [62]:Anything, [63]:Anything, [64]:Anything, [65]:Anything, [66]:Anything, [67]:Anything, [68]:Anything, [69]:Anything, [70]:Anything, [71]:Anything, [72]:Anything, [73]:Anything, [74]:Anything, [75]:Anything, [76]:Anything, [77]:Anything, [78]:Anything, [79]:Anything}
 LLVM view of erring value:   {} addrspace(10)** addrspace(11)* getelementptr inbounds ({ i64, {} addrspace(10)** }, { i64, {} addrspace(10)** } addrspace(11)* addrspacecast ({ i64, {} addrspace(10)** } addrspace(10)* bitcast ({} addrspace(10)* @"ejl_inserted$_Core_GenericMemory_24896$true$4744527424" to { i64, {} addrspace(10)** } addrspace(10)*) to { i64, {} addrspace(10)** } addrspace(11)*), i64 0, i32 1)

Stacktrace:
 [1] sum
   @ ./reducedim.jl:982
 [2] arr_to_num
   @ ./REPL[54]:1

Stacktrace:
  [1] sum
    @ ./reducedim.jl:982 [inlined]
  [2] arr_to_num
    @ ./REPL[54]:1
  [3] f
    @ ./REPL[55]:2 [inlined]
  [4] diffejulia_f_24869wrap
    @ ./REPL[55]:0 [inlined]
  [5] macro expansion
    @ ~/.julia/packages/Enzyme/i91IH/src/compiler.jl:6652 [inlined]
  [6] enzyme_call
    @ ~/.julia/packages/Enzyme/i91IH/src/compiler.jl:6131 [inlined]
  [7] CombinedAdjointThunk
    @ ~/.julia/packages/Enzyme/i91IH/src/compiler.jl:6015 [inlined]
  [8] autodiff
    @ ~/.julia/packages/Enzyme/i91IH/src/Enzyme.jl:521 [inlined]
  [9] autodiff
    @ ~/.julia/packages/Enzyme/i91IH/src/Enzyme.jl:542 [inlined]
 [10] gradient
    @ ~/.julia/packages/DifferentiationInterface/4n6vR/ext/DifferentiationInterfaceEnzymeExt/reverse_onearg.jl:303 [inlined]
 [11] gradient
    @ ~/.julia/packages/DifferentiationInterface/4n6vR/src/first_order/gradient.jl:63
 [12] shuffled_gradient
    @ ~/.julia/packages/DifferentiationInterface/4n6vR/src/first_order/gradient.jl:160 [inlined]
 [13] fwddiffe2julia_shuffled_gradient_25033wrap
    @ ~/.julia/packages/DifferentiationInterface/4n6vR/src/first_order/gradient.jl:0
 [14] macro expansion
    @ ~/.julia/packages/Enzyme/i91IH/src/compiler.jl:6652 [inlined]
 [15] enzyme_call
    @ ~/.julia/packages/Enzyme/i91IH/src/compiler.jl:6131 [inlined]
 [16] ForwardModeThunk
    @ ~/.julia/packages/Enzyme/i91IH/src/compiler.jl:6031 [inlined]
 [17] autodiff
    @ ~/.julia/packages/Enzyme/i91IH/src/Enzyme.jl:673 [inlined]
 [18] autodiff
    @ ~/.julia/packages/Enzyme/i91IH/src/Enzyme.jl:562 [inlined]
 [19] autodiff
    @ ~/.julia/packages/Enzyme/i91IH/src/Enzyme.jl:534 [inlined]
 [20] pushforward
    @ ~/.julia/packages/DifferentiationInterface/4n6vR/ext/DifferentiationInterfaceEnzymeExt/forward_onearg.jl:94 [inlined]
 [21] hvp
    @ ~/.julia/packages/DifferentiationInterface/4n6vR/src/second_order/hvp.jl:331 [inlined]
 [22] hvp(f::typeof(f), backend::AutoEnzyme{…}, x::Vector{…}, tx::Tuple{…}, contexts::Cache{…})
    @ DifferentiationInterface ~/.julia/packages/DifferentiationInterface/4n6vR/src/second_order/hvp.jl:75
 [23] top-level scope
    @ REPL[61]:1
Some type information was truncated. Use `show(err)` to see complete types.

Environment:

(@enz109) pkg> st
Status `~/.julia/environments/enz109/Project.toml`
  [a0c0ee7d] DifferentiationInterface v0.7.12
  [7da242da] Enzyme v0.13.109

julia> versioninfo()
Julia Version 1.11.8
Commit cf1da5e20e3 (2025-11-06 17:49 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: macOS (arm64-apple-darwin24.0.0)
  CPU: 11 × Apple M3 Pro
  WORD_SIZE: 64
  LLVM: libLLVM-16.0.6 (ORCJIT, apple-m2)
Threads: 1 default, 0 interactive, 1 GC (on 5 virtual cores)

[gdalle]

Here's the pure-Enzyme MWE:

using Enzyme

arr_to_num(x::AbstractArray) = sum(vec(x .^ 4) .* transpose(vec(x .^ 6)))

function f(x, c)
    c[1] = arr_to_num(x)
    return c[1]
end

function g(x, c)  # gradient
    dx = zero(x)
    dc = zero(c)
    autodiff(Reverse, f, Duplicated(x, dx), Duplicated(c, dc))
    return dx
end

function h(x, c, dx_batch)  # HVP
    dc_batch = map(dx -> zero(c), dx_batch)
    result = autodiff(Forward, g, BatchDuplicated(x, dx_batch), BatchDuplicated(c, dc_batch))
    return result
end

x = [3.0, 5.0];
dx_batch = ([1.0, 0.0], [0.0, 1.0]);
c = [0.0];

g(x, c) == gradient(Reverse, arr_to_num, x)[1]  # true

h(x, c, dx_batch)[1]  # fails on Enzyme v0.13.109

[wsmoses]

This concurrently implies that error in the DI integration tests, since all of them have continued to pass throughout (and still do)

[gdalle]

The DI integration tests I added in #2531 are a subset of the true DI+Enzyme tests, which run for an hour and a half. But yeah I'm also disappointed I didn't add this one, of course

[wsmoses]

because of the nested differentiation and broadcasting, this is somewhat complicated. can you reduce the complexity of the code being differentiated (broadcasting generates a ton of code internally)

[gdalle]

Here's a simpler function which has the same bug. Every attempt I made to remove the broadcasting (including the tedious tricks of #2071, or replacing it with map) removed the bug:

function arr_to_num(x::AbstractArray)
    a = 1 * x
    b = 1 * a
    return sum(b)
end

[wsmoses]

the sum and multiply here still get lowered into quite a large amount of code, given that the error comes from sum itself it seems quite necessary to disect into the inside of sum here. If you can inline / reduce it further, we can take a look.

[gdalle]

Hi @wsmoses! I removed any trace of mapreduce or broadcast, is this sufficient for you to work with?

MWE

using Enzyme

@inline function mydouble(x)
    # returns 2x
    y = similar(x)
    for i in eachindex(x, y)
        y[i] = 2 * x[i]
    end
    return y
end

@inline function myouterproduct(x, y)
    # returns x .* transpose(y)
    z = similar(x, length(x), length(y))
    for i in eachindex(x)
        for j in eachindex(y)
            z[i, j] = x[i] * y[j]
        end
    end
    return z
end

function arr_to_num(x::AbstractArray)
    a = mydouble(x)
    b = myouterproduct(a, x)
    return b[1]
end

function f(x, c)
    c[1] = arr_to_num(x)
    return c[1]
end

function g(x, c)  # gradient
    dx = zero(x)
    dc = zero(c)
    autodiff(Reverse, f, Duplicated(x, dx), Duplicated(c, dc))
    return dx
end

function h(x, c, dx_batch)  # HVP
    dc_batch = map(dx -> zero(c), dx_batch)
    result = autodiff(Forward, g, BatchDuplicated(x, dx_batch), BatchDuplicated(c, dc_batch))
    return result
end

let
    x = [3.0, 5.0]
    dx = ([1.0, 0.0],)
    dx_batch = ([1.0, 0.0], [0.0, 1.0])
    c = [0.0]
    h(x, c, dx)[1]  # works
    h(x, c, dx_batch)[1]  # fails
end

Error

ERROR: EnzymeRuntimeActivityError: Detected potential need for runtime activity.

Constant memory is stored (or returned) to a differentiable variable and correctness cannot be guaranteed with static activity analysis.
This might be due to the use of a constant variable as temporary storage for active memory (https://enzyme.mit.edu/julia/stable/faq/#faq-runtime-activity).
If Enzyme should be able to prove this use non-differentable, open an issue!

To work around this issue, either:
   a) rewrite this variable to not be conditionally active (fastest performance, slower to setup), or
   b) set the Enzyme mode to turn on runtime activity (e.g. autodiff(set_runtime_activity(Reverse), ...) ). This will maintain correctness, but may slightly reduce performance.


Mismatched activity for:   %.fca.0.7.insert = insertvalue { { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, i8*, {} addrspace(10)* addrspace(10)*, i64, {} addrspace(10)**, i8*, double*, double* }, double } %.fca.0.6.insert, {} addrspace(10)** %.sroa.19.0, 0, 7, !dbg !222 const val:   %.sroa.19.0 = phi {} addrspace(10)** [ %10, %L9.thread ], [ undef, %L96.loopexit ]
Type tree: {[0]:Pointer, [8]:Pointer, [16]:Pointer, [24]:Pointer, [32]:Pointer, [32,-1]:Float@double, [40]:Pointer, [48]:Integer, [49]:Integer, [50]:Integer, [51]:Integer, [52]:Integer, [53]:Integer, [54]:Integer, [55]:Integer, [56]:Pointer, [64]:Anything, [65]:Anything, [66]:Anything, [67]:Anything, [68]:Anything, [69]:Anything, [70]:Anything, [71]:Anything, [72]:Anything, [73]:Anything, [74]:Anything, [75]:Anything, [76]:Anything, [77]:Anything, [78]:Anything, [79]:Anything, [80]:Anything, [81]:Anything, [82]:Anything, [83]:Anything, [84]:Anything, [85]:Anything, [86]:Anything, [87]:Anything, [88]:Anything, [89]:Anything, [90]:Anything, [91]:Anything, [92]:Anything, [93]:Anything, [94]:Anything, [95]:Anything}
 LLVM view of erring value:   {} addrspace(10)** addrspace(11)* getelementptr inbounds ({ i64, {} addrspace(10)** }, { i64, {} addrspace(10)** } addrspace(11)* addrspacecast ({ i64, {} addrspace(10)** } addrspace(10)* bitcast ({} addrspace(10)* @"ejl_inserted$_Core_GenericMemory_13289$true$4738465344" to { i64, {} addrspace(10)** } addrspace(10)*) to { i64, {} addrspace(10)** } addrspace(11)*), i64 0, i32 1)

Stacktrace:
 [1] arr_to_num
   @ ~/Documents/GitHub/Julia/_Scratchpad/mwe2.jl:26

Stacktrace:
  [1] arr_to_num
    @ ~/Documents/GitHub/Julia/_Scratchpad/mwe2.jl:26
  [2] f
    @ ~/Documents/GitHub/Julia/_Scratchpad/mwe2.jl:30 [inlined]
  [3] diffejulia_f_13262wrap
    @ ~/Documents/GitHub/Julia/_Scratchpad/mwe2.jl:0 [inlined]
  [4] macro expansion
    @ ~/.julia/packages/Enzyme/QsBMf/src/compiler.jl:6652 [inlined]
  [5] enzyme_call
    @ ~/.julia/packages/Enzyme/QsBMf/src/compiler.jl:6131 [inlined]
  [6] CombinedAdjointThunk
    @ ~/.julia/packages/Enzyme/QsBMf/src/compiler.jl:6015 [inlined]
  [7] autodiff
    @ ~/.julia/packages/Enzyme/QsBMf/src/Enzyme.jl:521 [inlined]
  [8] autodiff
    @ ~/.julia/packages/Enzyme/QsBMf/src/Enzyme.jl:562 [inlined]
  [9] autodiff
    @ ~/.julia/packages/Enzyme/QsBMf/src/Enzyme.jl:534 [inlined]
 [10] g
    @ ~/Documents/GitHub/Julia/_Scratchpad/mwe2.jl:37 [inlined]
 [11] fwddiffe2julia_g_22680wrap
    @ ~/Documents/GitHub/Julia/_Scratchpad/mwe2.jl:0
 [12] macro expansion
    @ ~/.julia/packages/Enzyme/QsBMf/src/compiler.jl:6652 [inlined]
 [13] enzyme_call
    @ ~/.julia/packages/Enzyme/QsBMf/src/compiler.jl:6131 [inlined]
 [14] ForwardModeThunk
    @ ~/.julia/packages/Enzyme/QsBMf/src/compiler.jl:6031 [inlined]
 [15] autodiff
    @ ~/.julia/packages/Enzyme/QsBMf/src/Enzyme.jl:673 [inlined]
 [16] autodiff
    @ ~/.julia/packages/Enzyme/QsBMf/src/Enzyme.jl:562 [inlined]
 [17] autodiff
    @ ~/.julia/packages/Enzyme/QsBMf/src/Enzyme.jl:534 [inlined]
 [18] h(x::Vector{Float64}, c::Vector{Float64}, dx_batch::Tuple{Vector{Float64}, Vector{Float64}})
    @ Main ~/Documents/GitHub/Julia/_Scratchpad/mwe2.jl:43
 [19] top-level scope
    @ ~/Documents/GitHub/Julia/_Scratchpad/mwe2.jl:53

Environment

(@reac) pkg> st
Status `~/.julia/environments/reac/Project.toml`
  [7da242da] Enzyme v0.13.114

julia> versioninfo()
Julia Version 1.11.8
Commit cf1da5e20e3 (2025-11-06 17:49 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: macOS (arm64-apple-darwin24.0.0)
  CPU: 11 × Apple M3 Pro
  WORD_SIZE: 64
  LLVM: libLLVM-16.0.6 (ORCJIT, apple-m2)
Threads: 8 default, 0 interactive, 4 GC (on 5 virtual cores)
Environment:
  JULIA_EDITOR = code
  JULIA_VSCODE_REPL = 1
  JULIA_NUM_THREADS = 8

[gdalle]

Hi @wsmoses, do you need anything else from me here?
DI's tests have been failing for weeks because of this one bug, so if you have some cycles to fix it, I'd be very grateful

[gdalle]

Gentle bump

[wsmoses]

no I think this is sufficient for now, will poke if not