Dropping Requires-based extensions for Distributions, SpecialFunctions and StatsFuns?

[oschulz]

The @requires for Distributions, SpecialFunctions and StatsFuns turn LogarithmicNumbers from a lightweight package into an effectively quite heavy package, increasing the load time of packages depending on it (like @cscherrer's MeasureBase) to a degree that makes them unattractive as dependencies t themselves (I'd like to use MeasureBase.jl in BATBase.jl, for example, as part of bat/BAT.jl#351, but currently it's a too heavy because of LogarithmicNumbers' requires`).

It looks like Distributions & friends will not support LogarithmicNumbers directly (JuliaStats/Distributions.jl#1545):

@devmotion: I think it would be simpler if users that want to wrap the result of logpdf, logcdf etc. computations as a ULogarithmic use explicit calls of e.g. uexp(logpdf(dist, x)) etc. Load times could be improved for users of Distributions + LogarithmicNumbers by removing the overloads in LogarithmicNumbers and dropping the Requires dependency there.

@cjdoris, Would you consider dropping the @requires, as suggested by @devmotion?

[gdalle]

@cjdoris maybe a solution would be to switch to the lighter https://github.com/JuliaMath/DensityInterface.jl ?

[devmotion]

That would still lead to a LogarithmicNumber-specific API that is inconsistent with the original API in DensityInterface. IMO it seems also easier to just let users call uexp(logdensityof(...)) instead of something like densityof(ULogFloat64, ...).

[oschulz]

let users call uexp(logdensityof(...)) instead of something like logdensityof(ULogFloat64, ...)

I agree - it removes dependencies, is clear to read and just as compact.

[gdalle]

On the other hand I like the fact that logdensityof a logarithmic number works out of the box. This allowed me to implement a log-scale HMM estimation algorithm without changing anything to my code, simply by allowing LogarithmicNumbers as input

[devmotion]

AFAICT this is different from the issue here. If logpdf etc. is implemented generically enough in Distributions (which it should), calling logpdf(d, x) where x and/or the parameters of d are logarithmic numbers should just work. It should not be necessary for one package to know about the other. The current overloads discussed in this issue are of the form pdf(::Type{<:Logarithmic}, ...) and don't seem needed for this use case.

[gdalle]

Alright, my bad :)

[oschulz]

I like the fact that logdensityof a logarithmic number works out of the box

You don't mean logdensityof(logarithmic_number), right? Because that shouldn't work, logdensityof is supposed to take objects that are or have densities as it's argument, not actual density values. If you already have the number you would do log(logarithmic_number), if I understand correctly.

[gdalle]

Sure, I meant stuff like logdensityof(distribution, array_of_lognumbers)

[oschulz]

Sure, I meant stuff like logdensityof(distribution, array_of_lognumbers)

Oh, right, of course!

[cscherrer]

Thanks @oschulz , it would be great to have this package loading faster!

For some additional context, the core of the dependencies seems to be this:

const DISTRIBUTIONS_OVERLOADS = [f => Symbol(:log, f) for f in [:cdf, :ccdf, :pdf]]
const STATSFUNS_DISTS = [:srdist, :nchisq, :hyper, :ntdist, :tdist, :binom, :pois, :fdist, :norm, :beta, :nfdist, :chisq, :gamma, :nbeta, :nbinom]
const STATSFUNS_OVERLOADS = [Symbol(d, f) => Symbol(d, :log, f) for d in STATSFUNS_DISTS for f in [:cdf, :ccdf, :pdf]]
const SPECIALFUNCTIONS_OVERLOADS = [f => Symbol(:log, f) for f in [:gamma, :factorial, :beta, :erfc, :erfcx]]

This maps each function name f to a name for log ∘ f. I haven't measured, but I think StatsFuns and SpecialFunctions are pretty light-weight. And there definitely common enough that many people using LogarithmicNumbers would be using them already anyway.

So one possibility would be for this package to depend explicitly on StatsFuns and SpecialFunctions. But if the goal is to get rid of the @requires and Distributions will not add the dependency, we'd lose the DISTRIBUTION_OVERLOADS, which would be unfortunate.

One possibility of avoiding this would even allow this package to ignore StatsFuns and SpecialFunctions, and things could still "just work". Maybe there could be a very lightweight interface package [yet again, these just seem to solve everything] specifying higher-order functions f -> log ∘ f and f -> exp ∘ f. That would give any package a way to give semantic context relating these functions, similar to the approach in InverseFunctions.jl.

For example, if these are called logcirc and expcirc (just placeholder names, not suggesting these), then Distributions could depend on this proposed very light package and then add

logcirc(::typeof(pdf)) = logpdf
logcirc(::typeof(cdf)) = logcdf
logcirc(::typeof(ccdf)) = logccdf
expcirc(::typeof(logpdf)) = pdf
expcirc(::typeof(logcdf)) = cdf
expcirc(::typeof(logccdf)) = ccdf

[oschulz]

But if the goal is to get rid of the @requires and Distributions will not add the dependency, we'd lose the DISTRIBUTION_OVERLOADS, which would be unfortunate.

But why would that be so unfortunate? From what I understand, we'd only lose convenience definitions like pdf(ULogarithmic, d, x) - I think @devmotion makes a good point here, writing uexp(logpdf(d, x)) is not less compact and very clear.

[oschulz]

I haven't measured, but I think StatsFuns and SpecialFunctions are pretty light-weight. [...] So one possibility would be for this package to depend explicitly on StatsFuns and SpecialFunctions [...]

They are not light-weight, really - compare

julia> using InverseFunctions # Load a super-lightweight package to get some initial Pkg costs out of the way

julia> @time_imports using SpecialFunctions, StatsFuns, Distributions, LogarithmicNumbers, MeasureBase
    [...]
    226.6 ms  SpecialFunctions
    [...]
     92.7 ms  StatsFuns
    [...]
    470.4 ms  Distributions
    [...]
    379.7 ms  LogarithmicNumbers
    [...]
     74.8 ms  MeasureBase

with

julia> using InverseFunctions

julia> @time_imports using LogarithmicNumbers, MeasureBase
      0.4 ms  ┌ Requires
     37.1 ms  LogarithmicNumbers
    [...]
    287.2 ms  MeasureBase

So the load time of LogarithmicNumbers without requires is just about 40 ms, and the cost of MeasureBase on top of LogarithmicNumbers and it's other deps would just be 74.8. SpecialFunctions and Distributions are much heavier, and StatsFuns isn't exactly lightweight either - timed relatively on top of each other.

If LogarithmicNumbers were to depend on SpecialFunctions and StatsFuns, it's load time would be over 300 ms, while without @requires it can load in under 40 ms.

These times may not seem long, but larger use cases have many dependencies that add up, and suddenly you end up with load times of 10 seconds for a use case (and then I have to listen to people telling me "but in Python it would already be finished. :-) ).

[cjdoris]

Having read this discussion and the one over on Distributions, I think I agree that just removing all those overloads is the right thing to do. No registered packages use the behaviour.

[cscherrer]

This is a little off topic, but I'm generally opposed to making real design decisions for the benefit of people who think using JIT compilation for a short-running script is a good idea. That use case can benefit from static compiler work, otherwise it's a terrible cost model.

writing uexp(logpdf(d, x)) is not less compact and very clear.

Good point, I had missed this. So, maybe the requires-es can just go away? Or maybe there could be a way for a @load StatsFuns to call using StatsFuns and define the overloads. Even if it had to be called manually, that could be ok and might be a nice middle ground.

I guess I'll wait for @cjdoris to weigh in here, since it's his package and all :)

EDIT: Oops, I was typing and missed his comment ☝️

[cjdoris]

Btw uexp is not API, you do exp(ULogarithmic, x).