Fix promotion in cov and mean to handle integer and rational matrices #23285
Conversation
base/statistics.jl
Outdated
| scale!(unscaled_covzm(x, vardim), inv(size(x,vardim) - Int(corrected))) | ||
| function covzm(x::AbstractMatrix, vardim::Int=1; corrected::Bool=true) | ||
| C = unscaled_covzm(x, vardim) | ||
| T = typeof(first(C) / 1) |
There was a problem hiding this comment.
can unscaled_covzm ever return heterogeneously-typed arrays?
There was a problem hiding this comment.
It can. Are you worried about first not being representative of the elements in C?
There was a problem hiding this comment.
it should usually be a subtype of the element type, but other elements might be wider for abstractly typed arrays, right?
There was a problem hiding this comment.
The problem is that the element type has to be fairly concrete before we can instantiate it but I've just realized that in this case here, it should be possible to use promote_type(typeof(first(C) / 1), eltype(x)). I'll update.
There was a problem hiding this comment.
eltype(x) or eltype(C)? will unscaled_covzm ever result in a different eltype?
There was a problem hiding this comment.
Probably rare. I can only think of Bool entries and there is wouldn't change T but I think you are right that I should use C and not x
Fixes #8080