perf(LocalFeedRepository): speed up local feed extraction #7214
Conversation
|
I was just thinking that instead of waiting until all the sub-channels are extracted before updating the feed, we could update it as soon as new content is extracted from a channel. This would make the user feel that the feed is being updated more quickly, enhancing the overall UX. |
|
While I agree that immediately showing new items is a good idea, I doubt a good implementation is possible, as we don't know which channels have uploaded (and in which order), so the new entries would constantly be reshuffled, which I think would be rather confusing and possibly nauseating. Additionally, constantly adding new items at (essentially) random points in the recycler view might lead to performance problems. That aside, I don't see how this is related to this PR? I think it would be better to open a new issue. |
I agree that immediately showing the loaded videos ain't a good idea. Apart from the mentioned points it's also not very easy to implement with the currently used app architecture, as we still need some kind of compatibility with the Piped way of loading the feed. |
|
What's the reason for the current draft state? The PR seems ready to be merged at first glance, doesn't it? |
|
Code-wise yes, and I thought so too, but yesterday I noticed some rate limiting in playback. I also took a look at how RSS feed readers handle concurrent fetching, so I would like to adjust the values here again and test it in the next few days. |
FineFindus
force-pushed
the
feat/increase-batch-size
branch
from
ea27f8c to
f4ac17f
Compare
Speeds up the local feed extraction, by fetching more feeds concurrently and only applying a large timeout after fetching the full channels. This is based on the idea that YouTube's RSS feeds (extracted by the generic `FeedInfo`) have no/much less rate-limiting, likely due to the nature of RSS readers. Ref: TeamNewPipe/NewPipe#11817 feat(LocalFeedRepository): remove RSS delay
FineFindus
force-pushed
the
feat/increase-batch-size
branch
from
f4ac17f to
7f52592
Compare
|
|
||
| val totalExtractionCount = AtomicInteger() | ||
| val chunkedExtractionCount = AtomicInteger() | ||
| val channelExtractionCount = AtomicInteger() |
There was a problem hiding this comment.
We could possibly add some small comments about the different atomic integers and their difference here, but that's not that important
Bnyro
left a comment
Bnyro
left a comment
There was a problem hiding this comment.
Didn't experience any throttling while testing and the changes are certainly a UX improvement 👍
3 participants
Speeds up the local feed extraction, by fetching more feeds concurrently and only applying a large timeout after fetching the full channels. This is based on the idea that YouTube's RSS feeds (extracted by the generic
FeedInfo) have no/much less rate-limiting, likely due to the nature of RSS readers. I couldn't find an explicit statement from YouTube about that, but it makes sense. I've also increased to amount of concurrently fetched feeds in my fork for the last couple of months and had no noticeable impact so far.A small timeout after each chunk is still applied, to be a good citizen of the web and not overwhelm/spam the server.
The used values may need to be adjusted in the future, either to allow for even faster extraction or throttling if a rate-limit is introduced.