DeviantartExtractor - check archive sooner

[solo989]

With archives enabled the deviantartextractor does way too many api requests when looking through entire gallerys or folders.
If the archive was checked earlier these api requests could be avoided.
This isn't much of an issue on initial runs but becomes a problem on later runs when most of the files will have already been downloaded.

Most of the extensions should be able to be determined earlier and an additional archive entry could be added signifying that all the files from a particular url have already been downloaded. This would cover cases where there are both jpg and swf file links from one source.
The extension could be .finished or .done or maybe even something with characters that are invalid in real extensions.

On another note it would be nice if there were some way to specify the max resolution you want to download files in. This would could avoid additional api calls and lead to shorter download times.

[mikf]

I think what you are looking for is the extractor.skip option. Instead of going through all deviations on later runs, you can configure the DeviantartExtractor to stop when a certain number of already downloaded items are encountered.

The only "extra" API calls that happen from not checking the archive sooner and could be avoided is (possibly) 1 for each deviation to get the original download URL or journal content. All others - 1 initially to decide between user or group, and 1 per 24 deviations - are unavoidable.

This would cover cases where there are both jpg and swf file links from one source.

Extra swf or video files don't cause any extra overhead in terms of API calls. Their information is already included in a "basic" API response.

On another note it would be nice if there were some way to specify the max resolution you want to download files in. This would could avoid additional api calls and lead to shorter download times.

The resolution of the original file is unknown until its download URL is requested. Avoiding API calls like that wouldn't work.

[solo989]

I actually got it to work exactly how I wanted though my solution was a bit hacky and I ended up adding archive checks into deviantart.py.

The extra api calls are totally avoidable if what's checked from the archive can be derivated from the first api call. You can get the index from the url itself and the specific extension check could be bypassed by having the archive make the extension be .finished and have that be part of the check.
Obviously this should all be optional under a config option.

The extra files may not cause any extra api calls but if the jpg gets downloaded and then the program stops in the middle of downloading the swf file without an extra .finished file it would still do the full check again until it finishes downloading all the files and adds the .finished file to the archive.

Ideally the core code of the extractor baseclass would be updated to more seamlessly allow archive checks and adds in specific extractors and not just in the job.py handle_url.
Maybe even have handle_url itself return a dict that that gives you information on whether or not the download went through or not.

The resolution of the original file is unknown until its download URL is requested. Avoiding API calls like that wouldn't work.

The width and height are contained in content and if they are sufficient the additional api call can be avoided. Additionally you can do the extra api call and then compare the two different widths and heights and then choose to download the smaller one which would still reduce download time.

[kargaroc]

I don't know that much about this but randomized delays might possibly help too? The current system that just waits longer and longer is predictable and could be detected and blocked.

[phanduynam]

Has this issue been resolved?