Fix all tests and make everything work offline/with mocks by litetex · Pull Request #1332 · TeamNewPipe/NewPipeExtractor

litetex · 2025-07-11T19:31:08Z

Based on/Required before merging:

[YouTube] Add support for video lockup content type #1320
This should be SQUASHED when merging to not bloat the repo with mockfiles

List of fixes:

Made all tests mockable

Fixes #1276

You can now unplug your internet cable, run the tests with -Ddownloader=MOCK and everything will work
Tests in mock mode run in <5s in the IDE
Migrated data structure accordingly
Various minor improvements to downloader caching system
- Added REC alias for RECORDING downloader (so that I have to type less)
Tests that do network communication MUST implement InitNewPipeTest (or InitYoutubeTest for YT) so that the downloader is setup correctly
Added a few classes for abstraction like DefaultSimpleExtractorTest and DefaultSimpleUntypedExtractorTest
- This removes a TON (multiple hundred/thousand lines) of boilerplate code
Fixed: Some tests setup the downloader but never used it (particular LinkHandler-Tests)
Removed a bunch of region comments that are not region comments

YT

Fix Never Gonna Give You Up video having a different title (YoutubeMixPlaylistExtractorTest)
Disabled YoutubeMusicSearchExtractorTest - "showing results for ..." doesn't seem to be returned by the backend anymore, however the code is still present in the JS frontend.
- Added caching for YoutubeMusicSearchExtractor to not do the same work multiple times
Fix YoutubeStreamExtractorRelatedMixTest randomly failing:
Mix Videos can also display non-mix playlists in the related items:
~~Regenerated all YT mocks~~

Soundcloud

Top 50 list no longer exists (not present at https://soundcloud.com/charts) → Removed
User for SoundcloudChannelTabExtractorTest#Playlists no longer exists → Replaced with another user
- Chose a relatively big user that (also) makes copyright free music to ensure maximum test reliability
Fetch Charts in onFetchPage and not always when getInitialPage is called

Also:
Fixes #1283

I carefully read the contribution guidelines and agree to them.

TobiGr · 2025-07-13T09:12:26Z

Thank you. I was really needed to get more reliable tests and better detect flaws in PRs.

Top 50 list no longer exists (not present at https://soundcloud.com/charts) → Removed

Isn't it just called All music genres now? The thumbnails explicitly says TOP 50. All other charts are TOP 50 for a specific genre.

TobiGr

As already noted, these are relly good changes. I did not review SoundcloudChartsExtractor, but everything else related to the test changes which should be done by somebody else with more recent knowledge about the SoundCloud structure.

...r/src/test/java/org/schabi/newpipe/extractor/services/DefaultSimpleUntypedExtractorTest.java

...src/test/java/org/schabi/newpipe/extractor/services/youtube/YoutubeChannelExtractorTest.java

...rc/test/java/org/schabi/newpipe/extractor/services/youtube/YoutubeCommentsExtractorTest.java

...test/java/org/schabi/newpipe/extractor/services/youtube/YoutubeMixPlaylistExtractorTest.java

...rc/test/java/org/schabi/newpipe/extractor/services/youtube/YoutubePlaylistExtractorTest.java

...actor/src/test/java/org/schabi/newpipe/extractor/services/youtube/YoutubeSignaturesTest.java

.../java/org/schabi/newpipe/extractor/services/youtube/YoutubeStreamLinkHandlerFactoryTest.java

litetex · 2025-07-13T18:50:53Z

Top 50 list no longer exists (not present at https://soundcloud.com/charts) → Removed

Isn't it just called All music genres now? The thumbnails explicitly says TOP 50. All other charts are TOP 50 for a specific genre.

Oh I didn't notice that.

Update: Nope, that's not the Top 50 anymore they seems to have completely reworked how charts work.

There is also no more https://soundcloud.com/charts/top or https://soundcloud.com/charts/new just https://soundcloud.com/charts which then has links to various sets (aka playlists).

https://api-v2.soundcloud.com/charts?genre=soundcloud:genres:all-music&client_id=MoLbAg35TuqjYwWVtNIKyRPFScQGMOBY&kind=top (used for Top 50) returns nothing while the variant with kind=trending (for New & hot) still works

TobiGr · 2025-07-13T21:45:01Z

Update: Nope, that's not the Top 50 anymore they seems to have completely reworked how charts work.

Thanks for taking a closer look and sorry for causing confusion :)

extractor/build.gradle

litetex · 2025-07-14T18:43:06Z

Update: Nope, that's not the Top 50 anymore they seems to have completely reworked how charts work.

Thanks for taking a closer look and sorry for causing confusion :)

No problem :)
I created a new issue to track this: #1335

Forgotten comment

Rebased, should be gone now ^^

Stypox

Thank you! The commits are a bit messy but I guess there was no way to organize them better, given the huge amount of changes. I reviewed everything now and it seems ok. If any of my comments do not apply to the latest code, please ignore them, I could only review commits instead of the whole code as the browser would start lagging 😅

4277341 is only updating mocks, and that somehow fixed the test?
There are actually various SoundCloud trending categories here, though it's out-of-scope for this PR to implement them.

extractor/src/test/java/org/schabi/newpipe/extractor/GeneralNewPipeTest.java

extractor/src/test/java/org/schabi/newpipe/downloader/DownloaderFactory.java

...r/src/test/java/org/schabi/newpipe/extractor/services/DefaultSimpleUntypedExtractorTest.java

...rc/test/java/org/schabi/newpipe/extractor/services/bandcamp/BandcampSearchExtractorTest.java

...st/java/org/schabi/newpipe/extractor/services/media_ccc/MediaCCCRecentListExtractorTest.java

.../test/java/org/schabi/newpipe/extractor/services/peertube/PeertubeCommentsExtractorTest.java

...src/test/java/org/schabi/newpipe/extractor/services/youtube/YoutubeChannelExtractorTest.java

...actor/src/test/java/org/schabi/newpipe/extractor/services/youtube/YoutubeSignaturesTest.java

litetex · 2025-07-16T17:24:38Z

4277341 is only updating mocks, and that somehow fixed the test?

I'm not quite sure what happend there but I somehow got corrupted mocks, it didn't happen in subsequent runs so I think it might have been some sort of very weird race condition or YT just didn't like me xD

There are actually various SoundCloud trending categories here, though it's out-of-scope for this PR to implement them.

Belongs to #1335 ;)

litetex · 2025-07-16T17:44:28Z

@Stypox
Ty for the detailed review.

A few tests are currently failing due to the rebase I will have a look at them and after that (when the CI is green) we can probably merge/SQUASH if there is nothing else

Update: 2197 test passed, 108 ignored - We are good to go

"showing results for ..." doesn't seem to be returned by the backend anymore, however the code is still present in the JS frontend.

From review: * Fix format * Use ``extractorUrl`` name * YoutubeStreamLinkHandlerFactoryTest setup with no downloader Co-Authored-By: Tobi <[email protected]>

They no longer exists and the API returns nothing for them. Overall SoundCloud seems to use a new system.

We are not C / C++ devs: * https://medium.com/@muqsithirfan/java-null-comparison-fcfbcb55ac67

Also removed irrelevant methods See TeamNewPipe#1308

Co-Authored-By: Stypox <[email protected]>

These are currently not working, but the tests said OK because it was not checked what happens if the returned parameter is empty! -> TeamNewPipe#1339

litetex · 2025-07-16T19:37:08Z

There is currently a problem with n-Parameter deobfuscation, which is not working at all.
I created #1339 to handle this and disabled the test for now.

Stypox · 2025-07-17T07:05:50Z

.../test/java/org/schabi/newpipe/extractor/services/peertube/PeertubeCommentsExtractorTest.java

+        public void setUp() throws Exception {
+            super.setUp();
+
+            extractor(); // Initialize


What I meant here is to put extractor() directly in super.setUp() so we don't have to ever worry about the extractor not being initialized, or would that create issues? There are various other places where I saw // Init comments

What I meant here is to put extractor() directly in super.setUp() so we don't have to ever worry about the extractor not being initialized, or would that create issues?

Yes because there is no setUp method that creates an extractor when the class derives from InitYoutubeTest.

Also regarding "so we don't have to ever worry about the extractor not being initialized" there is a reason why this is lazily initialized:
We had a bunch of tests where there was a extractor used exactly in this way but it was never used.
When initializing the extractor during setup all tests mark it as used despite maybe never doing so.

There are various other places where I saw // Init comments

These are all places where this is used:

I now also refactored both classes to use dedicated methods which initialize the extractor.

TobiGr · 2025-07-18T06:23:11Z

This requires app changes / a migration when Top 50 kiosk is set as main page. Otherwise, you'll get this crash when starting the app:

Stacktrace

FATAL EXCEPTION: main
Process: org.schabi.newpipe.debug, PID: 7205
java.lang.NullPointerException: Attempt to read from field 'org.schabi.newpipe.extractor.linkhandler.ListLinkHandlerFactory org.schabi.newpipe.extractor.kiosk.KioskList$KioskEntry.handlerFactory' on a null object reference in method 'org.schabi.newpipe.extractor.linkhandler.ListLinkHandlerFactory org.schabi.newpipe.extractor.kiosk.KioskList.getListLinkHandlerFactoryByType(java.lang.String)'
	at org.schabi.newpipe.extractor.kiosk.KioskList.getListLinkHandlerFactoryByType(KioskList.java:146)
	at org.schabi.newpipe.fragments.list.kiosk.KioskFragment.getInstance(KioskFragment.java:81)
	at org.schabi.newpipe.settings.tabs.Tab$KioskTab.getFragment(Tab.java:357)
	at org.schabi.newpipe.settings.tabs.Tab$KioskTab.getFragment(Tab.java:313)
	at org.schabi.newpipe.fragments.MainFragment$SelectedTabsPagerAdapter.getItem(MainFragment.java:305)
	at androidx.fragment.app.FragmentStatePagerAdapterMenuWorkaround.instantiateItem(FragmentStatePagerAdapterMenuWorkaround.java:169)
	at androidx.viewpager.widget.ViewPager.addNewItem(ViewPager.java:1010)
	at androidx.viewpager.widget.ViewPager.populate(ViewPager.java:1158)
	at androidx.viewpager.widget.ViewPager.populate(ViewPager.java:1092)
	at androidx.viewpager.widget.ViewPager.onMeasure(ViewPager.java:1622)
	at android.view.View.measure(View.java:26496)
	at android.widget.RelativeLayout.measureChildHorizontal(RelativeLayout.java:735)
	at android.widget.RelativeLayout.onMeasure(RelativeLayout.java:481)
	at android.view.View.measure(View.java:26496)
	at android.view.ViewGroup.measureChildWithMargins(ViewGroup.java:6981)
	at android.widget.FrameLayout.onMeasure(FrameLayout.java:194)
	at android.view.View.measure(View.java:26496)
	at android.view.ViewGroup.measureChildWithMargins(ViewGroup.java:6981)
	at androidx.coordinatorlayout.widget.CoordinatorLayout.onMeasureChild(CoordinatorLayout.java:760)
	at androidx.coordinatorlayout.widget.CoordinatorLayout.onMeasure(CoordinatorLayout.java:833)
	at android.view.View.measure(View.java:26496)
	at androidx.drawerlayout.widget.DrawerLayout.onMeasure(DrawerLayout.java:1156)
	at android.view.View.measure(View.java:26496)
	at android.view.ViewGroup.measureChildWithMargins(ViewGroup.java:6981)
	at android.widget.FrameLayout.onMeasure(FrameLayout.java:194)
	at androidx.appcompat.widget.ContentFrameLayout.onMeasure(ContentFrameLayout.java:145)
	at android.view.View.measure(View.java:26496)
	at android.view.ViewGroup.measureChildWithMargins(ViewGroup.java:6981)
	at android.widget.LinearLayout.measureChildBeforeLayout(LinearLayout.java:1608)
	at android.widget.LinearLayout.measureVertical(LinearLayout.java:878)
	at android.widget.LinearLayout.onMeasure(LinearLayout.java:721)
	at android.view.View.measure(View.java:26496)
	at android.view.ViewGroup.measureChildWithMargins(ViewGroup.java:6981)
	at android.widget.FrameLayout.onMeasure(FrameLayout.java:194)
	at android.view.View.measure(View.java:26496)
	at android.view.ViewGroup.measureChildWithMargins(ViewGroup.java:6981)
	at android.widget.LinearLayout.measureChildBeforeLayout(LinearLayout.java:1608)
	at android.widget.LinearLayout.measureVertical(LinearLayout.java:878)
	at android.widget.LinearLayout.onMeasure(LinearLayout.java:721)
	at android.view.View.measure(View.java:26496)
	at android.view.ViewGroup.measureChildWithMargins(ViewGroup.java:6981)
	at android.widget.FrameLayout.onMeasure(FrameLayout.java:194)
	at com.android.internal.policy.DecorView.onMeasure(DecorView.java:760)
	at android.view.View.measure(View.java:26496)
	at android.view.ViewRootImpl.performMeasure(ViewRootImpl.java:4042)
	at android.view.ViewRootImpl.measureHierarchy(ViewRootImpl.java:2658)
	at android.view.ViewRootImpl.performTraversals(ViewRootImpl.java:2964)
	at android.view.ViewRootImpl.doTraversal(ViewRootImpl.java:2371)
	at android.view.ViewRootImpl$TraversalRunnable.run(ViewRootImpl.java:9297)
	at android.view.Choreographer$CallbackRecord.run(Choreographer.java:1231)
	at android.view.Choreographer$CallbackRecord.run(Choreographer.java:1239)
	at android.view.Choreographer.doCallbacks(Choreographer.java:899)
	at android.view.Choreographer.doFrame(Choreographer.java:832)
	at android.view.Choreographer$FrameDisplayEventReceiver.run(Choreographer.java:1214)
	at android.os.Handler.handleCallback(Handler.java:942)
	at android.os.Handler.dispatchMessage(Handler.java:99)
	at android.os.Looper.loopOnce(Looper.java:201)
	at android.os.Looper.loop(Looper.java:288)
	at android.app.ActivityThread.main(ActivityThread.java:7924)
	at java.lang.reflect.Method.invoke(Native Method)
	at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:548)
	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:936)

Fix all tests and make everything work offline/with mocks. Remove old mocks and generate new ones with new structure. Remove SoundCloud "Top 50" kiosk.

Stypox · 2025-10-04T14:44:08Z

Note: SoundCloud's clientId is stored statically and thus fetched only in one test. However, if that test is not executed again before all others, the clientId will not be set and other tests will try to obtain it with a call to https://soundcloud.com, but will fail because they have no mock request for that.

litetex force-pushed the fix-all-tests branch from 2450e21 to 3c2b9a1 Compare July 12, 2025 10:48

litetex mentioned this pull request Jul 12, 2025

[YouTube] Add support for video lockup content type #1320

Merged

7 tasks

litetex force-pushed the fix-all-tests branch from 3c2b9a1 to 40d0f9d Compare July 12, 2025 11:30

litetex changed the title ~~Fix all tests / Update mocks~~ Fix all tests and make everything work offline Jul 12, 2025

litetex marked this pull request as ready for review July 12, 2025 13:12

litetex changed the title ~~Fix all tests and make everything work offline~~ Fix all tests and make everything work offline/with mocks Jul 12, 2025

litetex mentioned this pull request Jul 12, 2025

[YouTube] Check if videos with multiple uploaders are handled correctly #1333

Closed

TobiGr added soundcloud service, https://soundcloud.com/ youtube service, https://www.youtube.com/ codequality Improvements to the codebase to improve the code quality tests Issues and PR related to unit tests labels Jul 12, 2025

TobiGr reviewed Jul 13, 2025

View reviewed changes

litetex force-pushed the fix-all-tests branch from 73ab92d to f6c16f7 Compare July 13, 2025 19:03

litetex mentioned this pull request Jul 13, 2025

Validate HTTP response codes across the extractor #1322

Open

3 tasks

TobiGr reviewed Jul 13, 2025

View reviewed changes

extractor/build.gradle Outdated Show resolved Hide resolved

litetex mentioned this pull request Jul 14, 2025

[Soundcloud] Charts rework? #1335

Open

litetex force-pushed the fix-all-tests branch from 227cd7b to 1b39226 Compare July 14, 2025 18:42

Stypox force-pushed the fix-all-tests branch 2 times, most recently from 8f2c322 to 1be3bc7 Compare July 16, 2025 15:02

Stypox requested changes Jul 16, 2025

View reviewed changes

litetex added 4 commits July 16, 2025 19:52

Fix YoutubeDashManifestCreatorsTest

e9a5d62

Fix YoutubeMixPlaylistExtractorTest#Music and update mocks

4f9e2a9

YoutubeMusicSearchExtractor: Add caching to improve performance

084e66a

Disable YoutubeMusicSearchExtractor test

9ef3a6e

"showing results for ..." doesn't seem to be returned by the backend anymore, however the code is still present in the JS frontend.

litetex and others added 13 commits July 16, 2025 19:52

Improved code style

89a0923

From review: * Fix format * Use ``extractorUrl`` name * YoutubeStreamLinkHandlerFactoryTest setup with no downloader Co-Authored-By: Tobi <[email protected]>

[SoundCloud] Purged Top 50

1b3a3f3

They no longer exists and the API returns nothing for them. Overall SoundCloud seems to use a new system.

Fix format

99b9cdb

We are not C / C++ devs: * https://medium.com/@muqsithirfan/java-null-comparison-fcfbcb55ac67

Modify SoundcloudChannelTabExtractorTest so that it compiles

8bbb7e9

Also removed irrelevant methods See TeamNewPipe#1308

Add mockdata for new SoundcloudChannelTabExtractorTest

60c7790

Improve naming

599bc78

Co-Authored-By: Stypox <[email protected]>

Fix typo

1621336

Co-Authored-By: Stypox <[email protected]>

Update docs

9a74872

Co-Authored-By: Stypox <[email protected]>

Init in setup

2d3a7ae

Co-Authored-By: Stypox <[email protected]>

Cleanup

066553d

Co-Authored-By: Stypox <[email protected]>

Fix naming

de96e98

Co-Authored-By: Stypox <[email protected]>

Updated mocks

c637c19

Fix broken real test and migrate mock data

c443874

litetex force-pushed the fix-all-tests branch from 56f4aa8 to c443874 Compare July 16, 2025 18:19

litetex added 2 commits July 16, 2025 20:20

Removed unused mocks

dd0f955

Improve n parameter tests

4fa9d74

These are currently not working, but the tests said OK because it was not checked what happens if the returned parameter is empty! -> TeamNewPipe#1339

Stypox requested changes Jul 17, 2025

View reviewed changes

Use dedicated methods to fetch extractor related data

3f0b055

Stypox approved these changes Jul 18, 2025

View reviewed changes

TobiGr approved these changes Jul 18, 2025

View reviewed changes

TobiGr merged commit 404f503 into TeamNewPipe:dev Jul 18, 2025
4 checks passed

TobiGr mentioned this pull request Jul 18, 2025

Update Extractor and add migration to remove SoundCloud Top 50 kiosk TeamNewPipe/NewPipe#12438

Merged

2 tasks

litetex deleted the fix-all-tests branch July 18, 2025 20:38

TobiGr mentioned this pull request Jul 21, 2025

[YouTube] Mark members-only videos #1280

Merged

3 tasks

TobiGr mentioned this pull request Oct 11, 2025

Fix SoundCloud channel tab pagination looping on empty pages #1387

Merged

3 tasks

Uh oh!

Conversation

litetex commented Jul 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Made all tests mockable

YT

Soundcloud

Uh oh!

TobiGr commented Jul 13, 2025

Uh oh!

TobiGr left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

litetex commented Jul 13, 2025

Uh oh!

TobiGr commented Jul 13, 2025

Uh oh!

Uh oh!

litetex commented Jul 14, 2025

Uh oh!

Stypox left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

litetex commented Jul 16, 2025

Uh oh!

litetex commented Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

litetex commented Jul 16, 2025

Uh oh!

Stypox Jul 17, 2025

Choose a reason for hiding this comment

Uh oh!

litetex Jul 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TobiGr commented Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Stypox commented Oct 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

litetex commented Jul 11, 2025 •

edited

Loading

TobiGr left a comment •

edited

Loading

Stypox left a comment •

edited

Loading

litetex commented Jul 16, 2025 •

edited

Loading

litetex Jul 17, 2025 •

edited

Loading

TobiGr commented Jul 18, 2025 •

edited

Loading