read_excel progress bar

[bbrewington]

It would be nice to have a progress bar like readr.

Here's the "show_progress" function from https://github.com/tidyverse/readr/blob/8d6a892b941948c96b6a5aa2ddb5b897492478e3/R/utils.R

show_progress <- function() {
!isTRUE(getOption("readr.show_progress")) || # user disables progress bar
!interactive() || # not an interactive session
!is.null(getOption("knitr.in.progress")) # Not actively knitting a document
}

[jennybc]

I'm curious. With what size sheet are you really feeling this need? xlsx or xls?

[jennybc]

Do as @jimhester plans to do in readr (tidyverse/readr#611) and use

https://github.com/gaborcsardi/progress

[bbrewington]

I think it ended up being a corrupt xls file...oops :) Are you thinking it should read fast enough that a progress bar isn't necessary?

[jennybc]

@bbrewington I just wanted a data point from the real world re: at what size progress starts to seem necessary. The sheets I work with during development are weird but small.

[jennybc]

We're not going to implement progress bar at this point. Not clear the work is justified by the pay off.

[bluunk]

One data point: I am using readxl to open files with a size of >100 MB. This takes quite long. Thus, some feedback in form of a progress bar would be helpful, just to know if R is still working or got stuck

[jennybc]

I'll re-open this so I take another look at the next release.

[jennybc]

I have a progress indicator now in a local branch but could use help from actual users. I am interested in:

• At what size is readxl::read_excel() slow enough that you would even want a progress indicator?
  • size of overall .xls or .xlsx, dimensions of a worksheet that is slow to read
• What is your OS and how much RAM?
• Do you know of any inconveniently large publicly available xls or xlsx files?

[nacnudus]

This xlsx is the largest I know, 25M overall with a sheet 211,933 rows by 10 columns. But it loads in a second or so on my Linux laptop with 16GB RAM and I wouldn't want a progress indicator until it was maybe three seconds.

That site seems to have trouble, so I could email it instead.

[jennybc]

I have a .xls that's 117MB (many different worksheets) but, like @nacnudus's experience, the progress bar barely has time to display before read_excel() completes.

[scjs]

The Voice_Master.xlsx spreadsheet here (the first link, "Acoustics") is 73 MB with 16k rows and about 500 columns in one worksheet. It is CC BY-NC-SA 3.0 licensed. Loading it with read_excel() takes about 20 seconds on Windows 7 with 8 GB RAM. I used openxlsx to convert and clean it in this document.

[jennybc]

Wow @scjs that is a great find. Not only does it take a non-trivial time to read, but it also (probably related) demonstrates the over-messaging problem from #361. It hurts, but that's a great example!