{DeviantArt} Limit Number of API Info Retrievals

[Ssurgul]

I'm having to not be able to check first again because I'm not sure how to phrase this inquiry.

Is there a config option to limit the number of return headers on one of the API extractors? I listed it for DA above because it's the one I know about, but I'm sure there could be others that might apply here.

What I'm trying to do is to reduce, sharply, the amount of time needed for the API. In DA, there are several accounts that I have marked for downloading that have literally 10s of thousands of media entries. I already set the duplicate option to 5, but it seems like the API call must first download the list of every single item first. If it could be set to only get 100 entries at a time, that would definitely accelerate the download delays. (I'm currently using Task Til Dawn to automate the firing off of the various extracts/sites. But because of how long it takes to just start comparing the returned URLs, I had to break it into 3 different jobs. At least two of those jobs take well over 14 hours each time it runs.)

Any insights would be greatly appreciated.

Ssurgul

[mikf]

#8472 doesn't help in this situation?

---

I already set the duplicate option to 5

There is no "duplicate" option.

but it seems like the API call must first download the list of every single item first

It doesn't. gallery-dl processes items on the fly.

[Ssurgul]

Apologies, not the 'duplicate' but the skip option is already set.

As to processing things on the fly, I can't see how that's working with the question. If I have an account to update (already did the preliminary download), and it has only about 500 entries, with 6 new items, that takes only a couple of minutes. If I'm on the 20K account, and there are only 25 new items, it takes upwards of an hour or more before it starts downloading anything.

[mikf]

Do you have any "special" options enabled like folders?
It should start downloading immediately with default settings.

[Ssurgul]

Well, I specifically turn folders off, and I have the flat set as true, just to be sure.

Here's the relevant portion:

{
    "extractor":
	{
				
		"deviantart":
        {
			"include" : "gallery,scraps",
			"flat" : "true",
			"folders" : "false",
			"skip" : "abort:5",
			"sleep-request" : "6.00",
            "client-id" : SEKRIT,
            "client-secret" : SEKRIT #2,
			"base-directory" : "D:/Gallery-DL/DeviantArt/",
			"archive" : "D:/Gallery-DL/DeviantArt/DeviantArt.sqlite3",
			"directory" : ["{username}"],
			"filename" : "{published_time} {filename}.{extension}",
			"metadata" : "true",
			"videos" : "yt-dlp",
			"postprocessors": 
			{
                "name" : "metadata/custom@post",
                "content-format" : "{description}",
                "filename" : "{published_time} {title}.txt"
			}
        }
		
    },
	
    "downloader": 
	{
 		"ytdl":
		{
			"module":"yt-dlp"
		}
    },

[Ssurgul]

Ok, so on a whim, I removed the folders and flat config variables, and tried again with a 15K item account. This time it only took a couple of minutes, start to finish. So, clearly those items just can't be present at all.

[mikf]

"folders" : "false",

That's the problem.

This enables the folders option, as "false" is a non-empty string and is therefore interpreted as true. The string "false" is different than the boolean false.

[Ssurgul]

Closing topic as the fix is to remove the variables mentioned above.