Improving check_model() for GLMs

[bwiernik]

The current selection of plots returned by check_model() for GLMs aren't ideal in a few ways.

1. They are missing a linearity check (fitted vs residuals). For binomial models, this should be a called to binned_residuals(). For other families, the standard check is fine.
2. For binomial models, the constant variance plot should be omitted.
3. For binomial models, the residual QQ plot is hard to interpret.
4. For non-bernoulli models, we should include a plot for checking overdispersion.

For the latter few points, the DHARMa package provides an easy-to-interpret approach for checking distributional assumptions from qq plots and problems with fitted vs residual plots using quantile residuals. We might consider soft-importing DHARMa or re-implementing those approaches.
https://cran.r-project.org/web/packages/DHARMa/vignettes/DHARMa.html
florianhartig/DHARMa#33

[bbolker]

One standard check for bernoulli models is to do a binned binomial proportions plot (this is, loosely, a graphical equivalent of the Hosmer-Lemeshow test, and is implemented in arm::binnedplot()

[bwiernik]

Yep, that's implemented in performance as binned_residuals()

[bbolker]

ooops, I missed your comment about binned_residuals(). D'oh!

[bwiernik]

No worries! Thanks for your comment!

[IndrajeetPatil]

To be fair, though, it is kind of difficult to know the existence of this visual check for two reasons:

• Most users access the available checks via the check_model() output, but we don't include it there (as you mention in your original issue):

library(performance)

model <- glm(vs ~ wt + mpg, data = mtcars, family = "binomial")

x <- check_model(model)
names(x)
#> [1] "VIF"         "QQ"          "HOMOGENEITY" "OUTLIERS"    "INFLUENTIAL"

• As opposed to the majority of the other checks, the users need to call print method, instead of plot, but using plot actually works and returns incorrect plot (cf. print vs plot methods see#119)

print(binned_residuals(model))
#> Warning: Probably bad model fit. Only about 50% of the residuals are inside the error bounds.

plot(binned_residuals(model))
#> Warning: Probably bad model fit. Only about 50% of the residuals are inside the error bounds.

^{Created on 2021-12-09 by the reprex package (v2.0.1)}

[DominiqueMakowski]

Should we incorporate the plot into check_model?

[bwiernik]

Yes, like I said in the first post, I think we need to change most of the plots in check_model() for GLMs.

[bwiernik]

I think we also might want to add a check_predictions() plot to check_model() for all models, perhaps dropping the normal density plot?

[bbolker]

This isn't the place, but I may have some Opinions about binned_model (I would like a smoothed line added, binomial CIs on the points; I would rather have binomial CIs on the individual points and a confidence interval on the smooth rather than have a dichotomizing region that indicates 'good' and 'bad' points ...). I see that this is copying what arm::binnedplot does though ...

[bwiernik]

I like that. Have an example?

[bbolker]

Not yet but could do one tomorrow.

[bwiernik]

That would be great!

[jebyrnes]

Would it be easier to pull in https://github.com/florianhartig/DHARMa as its very flexible in terms of model type?

[bbolker]

https://gist.github.com/bbolker/945e684dc148239364ab0f99c9a1beed

I realized that the residuals from binomials are not binomial (so constructing binomial CIs isn't useful); however I think that a binned plot using mean_cl_boot() to generate CIs is probably fine. See examples and decide what you think is useful or not ...