Decide what to do about `pushforward` and co.

[ToucheSir]

We get at least a couple of confused questions about it every month, but AFAICT there isn't much appetite to bring it up to a point of general usefulness. How does a big flashing banner saying "don't use this" sound until if/when someone feels the urge to pick it back up?

[DhairyaLGandhi]

We added the forward mode to be used, and it's actually the fastest forward mode on neural networks right now, so I would say if we get questions we should help resolve them and flesh it out.

[ToucheSir]

Are there benchmarks floating out there that demonstrate this? I was under the impression that rule support was not fleshed out enough to support all but the most basic of Flux models.

[DhairyaLGandhi]

We can do mlps which are of most importance to small model cases such as SciML. Small - medium sized models are also the ones that are poised to gain most from a fast forward mode, beyond which a reverse mode would start making sense, esp where a lot of arguments are required.

Also see #503 (comment)

[ToucheSir]

Thanks for the link, now I'm curious why SciML hasn't picked it up for the diffeq(flux) ecosystem. I'm sure there's a perfectly legitimate reason, but I didn't see any discussions about it outside of SciML/NeuralPDE.jl#150. Perhaps @ChrisRackauckas recalls the background there?

Also, we could definitely use some documentation for this feature if it's going to be sticking around. Currently, there are a grand total of 0 doc comments on https://github.com/FluxML/Zygote.jl/tree/master/src/forward, and hence no docs section on the Zygote site either. I'm sure there are PRs somewhere that address this, but when I checked yesterday everything since #503 has either a) been in limbo for over a year or b) closed outright?

[ChrisRackauckas]

No docs and some things ran into issues really quickly. The conclusion was just, wait for Diffractor.

[gdalle]

What is the status of this in 2024? @oxinabox any insights? I just had a question asking whether we should support Zygote's forward mode in DI

[oxinabox]

I have never heard of anyone using it.
I have never used it.
It was one of the last things Mike did before leaving AFAIK.