feat(StreamsExtractor): generate PoToken

[FineFindus]

Implements support for locally generating PoTokens using the device web view. This is a direct port of
TeamNewPipe/NewPipe#11955 to native Kotlin.
However, there are some notable differences:

• NewPipe use an additional PoTokenGenerator interface. As there is only one implementation right now, this is skipped
• NewPipe handles broken web views on older devices. Since we're increasing minSdk in BREAKING CHANGE: bump minSdk to 26 #7047, I'm hoping we can get away without

It is marked as a draft, since while it does generate valid PoTokens, playback does not work (Source error: open failed: ENOENT; help is appreciated).

Closes: #7065

[FineFindus]

Please don't add unnecessary comments. Linking an issue to be closed is fine, but the rest just adds noise. I'm already well aware of the NewPipe PR (see description above).

[Figim]

Please don't add unnecessary comments. Linking an issue to be closed is fine, but the rest just adds noise. I'm already well aware of the NewPipe PR (see description above).

Sorry.

1. "Source Error" only occurs if "HLS" is enabled. DASH works great. tested

2. Live streams also play smoothly on HLS

[Bnyro]

The crash log is a temporary DNS issue on your side, that's not LibreTube related.

[Figim]

A source error still occurs when starting a video in HLS.

XRecorder_20250205_01.mp4

[FineFindus]

A few questions:

• Since HLS does not work with potokens either, should it be removed/disabled?
• JavaScriptUtil.kt uses nanojson, should it be ported as well? I left it for now, as it would make it easier to follow future changes. There isn't any noticable impact on filesize, since it is already included in NewPipeExtractor.

[YT-Advanced]

• Since HLS does not work with potokens either, should it be removed/disabled?

This issues seem to be NewPipeExtractor bug itself, since HLS don't need pot but it still be appended in streaming url

• JavaScriptUtil.kt uses nanojson, should it be ported as well? I left it for now, as it would make it easier to follow future changes.

Uhm u should port it also

[gechoto]

With the current code the purpose of JavaScriptUtil.kt is not perfectly clear at first look and it might take a while to notice that this is for po tokens.
The class and function names are pretty generic.
For example in the first function it says something about some "challenge data". Yeah cool, which challenge data? Someone new to the project won't know.

I suggest to do one of these to make it more obvious it is about po tokens:

• Move JavaScriptUtil.kt and PoTokenWebView.kt to a sub-package named potoken
• Rename JavaScriptUtil.kt to something like PoTokenJavaScriptUtil.kt
• Rename the functions and extend the documentation in JavaScriptUtil.kt to make it clear this is for po tokens

NewPipe did the first option here and put everything po token related into a sub package:
https://github.com/TeamNewPipe/NewPipe/tree/dev/app/src/main/java/org/schabi/newpipe/util/potoken

[Bnyro]

Since HLS does not work with potokens either, should it be removed/disabled?

It shouldn't be completely removed because it might still work with some private Piped instances, however disabling it by default is possible.

JavaScriptUtil.kt uses nanojson, should it be ported as well? I left it for now, as it would make it easier to follow future changes.

Yes, I would like to keep the used libraries consistent across the code. Since the NanoJSON code part is only very small and the code differs already due to the fact that it's ported to Kotlin, I think it's reasonable to port it.

I don't have much time to look into the problems you mentioned at the moment (and the next few weeks) unfortunately due to tons of exams at university. I'll see if I can find some time to allocate for reviewing soon, but no guarantees.

[FineFindus]

Disabled and ported.

I don't have much time to look into the problems you mentioned at the moment (and the next few weeks) unfortunately due to tons of exams at university. I'll see if I can find some time to allocate for reviewing soon, but no guarantees.

Sure, no pressure. I would also like to clean up the commits before merging. Good luck with your exams.

[gechoto]

There should be some better error handling.
Currently if something in parseChallengeData or parseIntegrityTokenData fails the app crashes.

[gechoto]

There is a NewPipe PR which ports po token code to coroutines

TeamNewPipe/NewPipe#12028

Maybe this could be used as a base.
With the code in the linked PR the app does not crash.
It does handle exceptions better.

[FineFindus]

Currently if something in parseChallengeData or parseIntegrityTokenData fails the app crashes.

Please attach a crash log/reproducible case and don't post vague comments about 'something' crashing.

[gechoto]

Currently if something in parseChallengeData or parseIntegrityTokenData fails the app crashes.

Please attach a crash log/reproducible case and don't post vague comments about 'something' crashing.

One specific crash log won't be useful. It crashes on any error.
Just do something obviously wrong in parseChallengeData, could be corrupted challenge data or for example try to access something which does not exist:

val messageId = challengeData[10].jsonPrimitive.content

[gechoto]

since you access challengeData null-safe now
ae23ba3

this could also be applied to these two

but I don't think this is a perfect solution
because there are still other things which can go wrong and make the app crash

accessing something on challengeData out of bounds was only one example

there should be better error handling in general instead
maybe this whole parseChallengeData should be ran in try/catch

there is this NewPipe PR now which seems to handle errors better:
TeamNewPipe/NewPipe#12028

[gechoto]

idk maybe it is better to not use getOrNull here and instead log an exception

problem is that currently LibreTube crashes instead of logging the exception and showing the user an error message
all of this stuff should be wrapped in an error handler
not just these few lines

from my experience integrating po token support in OuterTune error handling works better with the changes from this PR:
TeamNewPipe/NewPipe#12028

[Bnyro]

First of all, thank you for all your work spent into this PR! Everything works very nicely, the only downside is the increased loading time (as you mentioned), but that's certainly worth it.

About my review:

• I've added/pushed some small changes to abuse the Kotlin syntax, please feel free to just squash them if they're okay
• I'm a bit worried about the GOOGLE_API_KEY and the SECRET_KEY used here? I assume they are from NewPipe? If we hardcode them, this might make it easy for YouTube to fingerprint our client because they are publicly visible and YouTube could just block all request using them. That's nothing blocking this PR, it's just a side note that we maybe could consider making a build option for providing these keys in the future (if obtaining them is documented somewhere).
• I've left some other small nitpicks/questions, feel free to answer or just ignore/resolve them if they're irrelevant

With that being mentioned, I think that there's nothing in the way from merging this and creating a release soon after (if I find time for it).

[Bnyro]

@gechoto I agree on you that we could still improve the errror handling here in the future, but that's nothing important enough to block this PR and could still be done with follow-up pull requests.

[FineFindus]

I'm a bit worried about the GOOGLE_API_KEY and the SECRET_KEY used here? I assume they are from NewPipe?

They are, but I doubt there is a way without using them, since they're required for the request. They have been in use since at least 2023 and are included in dozens of repos at this point, so I doubt there is any use in hiding them now :)

[Bnyro]

Thanks for looking into the review and rebasing!