Introduce `DifferentiationInterface.jl` for Jacobian computation

[SouthEndMusic]

Quite a while back we put a lot of effort in getting ForwardDiff.jl to work as an automatic differentiation (AD) method for computing Jacobians for implicit solvers (see e.g. #550). Then after a while we decided to make finite difference the standard differentiation method again (see #1832).

This is a pity, because AD should in general be more accurate and AD is very well supported in the SciML ecosystem. The newest efforts there are for supporting DifferentiationInterface.jl, which should make it as easy to switch between different AD backends as it is now already to switch between different solver algorithms.

One of the things that makes AD support complex is (pre)allocating the right caches for intermediate results. DifferentiationInterface.jl attempts to automate this but that is not always possible when computations are in-place (as ours are). We currently use PreallocationTools for this, which is somewhat intrusive in our codebase, quite magical to the uninitiated and is quite fiddly to get to work well together with continuous control types, i.e. ContinuousControl and PidControl. Therefore it would be nice of we could use an AD backend that does not require this 'manual' caching, for which Enzyme.jl is a promising candidate that is gaining popularity in the SciML community.

One potential hurdle with Enzyme.jl is that it might make our current executable significantly bigger, but that is all the more reason to work towards static compilation 🙂

[visr]

Yes, nice writeup. I assume this would need SciML/OrdinaryDiffEq.jl#2567? Which luckily seems quite close.

[SouthEndMusic]

Ah, I wasn't aware of that PR. We don't necessarily have to wait for that to investigate, as we can pass our own Jacobian function (which @SvenWesterbeek knows all about ^^) using the DI API for in-place Jacobians directly

[SvenWesterbeek]

Happy to share my findings if you ever need it @SouthEndMusic :)

[gdalle]

Just popping by to say that I'll be happy to help you integrate DI into your package. As far as preallocation is concerned, are you aware of DI.Cache (see https://juliadiff.org/DifferentiationInterface.jl/DifferentiationInterface/stable/explanation/advanced/#Contexts)? This could possibly be a first step towards getting rid of PreallocationTools. At the moment, these caches are not yet optimized because no one has asked me, but if you need them I'll get on that.