[feature request] conditional `part-directory`

[powerfulsaber]

I want to use a different path for part-directory depending on the size of the file. I currently have it set to a directory under /tmp/ for various reasons, but it can cause issues when the file is very large and/or running multiple instances.

Also, please expose temppath to the exec postprocessor, so execs running at event file can use it. I did it myself by adding kwdict["_temppath"] = pathfmt.temppath after line 71 of gallery_dl/postprocessor/exec.py.

[mikf]

part-directory now supports selecting a path by conditions: 4e2987e
It usually doesn't have access to a file's size, though.

Accessing _temppath is also supported, by the way: ca4e061

[powerfulsaber]

I've been using my own implementation that is similar, but it decides part-directory in http.py, so it's always able to get the file size. I haven't noticed any issues for the time I've been using it, and it passes all the tests.

I can create a pull request if it would help.

[mikf]

Well, yeah, it is usually possible to get the file size after making a connection via the Content-Length response header, but .part file downloads are meant to be resume-able, meaning that you need its size before sending a request to send the appropriate Range header, so part-directory needs to be evaluated and applied beforehand.

[powerfulsaber]

Yes, so I decided to make a trade-off by having it make an additional request with the correct range headers afterward. I'll create a PR so you can evaluate.

[mikf]

by having it make an additional request

No sorry, not happening. At least not by default.

I forgot to mention that this also needs to work for the ytdl downloader, where getting the file size is even more involved or outright impossible.

[powerfulsaber]

It only does so if a conditional partdir is used and an existing part file is detected, so it won't occur by default. And I did make an attempt for the ytdl downloader, but it's currently not working.

[powerfulsaber]

If I can get the ytdl portion working, would you be open to it?

[mikf]

No. This is far too much effort and code for what I would consider a "fringe" use case. I don't want to merge all of this into the already overloaded http downloader code for a feature that almost nobody is going to use.