Skip to content

Conversation

@pblazej
Copy link
Contributor

@pblazej pblazej commented Oct 2, 2025

Let's try to separate on-device tests from the core suite (with proper retry, timeout, etc.).

@github-actions
Copy link

github-actions bot commented Oct 2, 2025

⚠️ This PR does not contain any files in the .changes directory.

@pblazej
Copy link
Contributor Author

pblazej commented Oct 2, 2025

@hiroshihorie you can disable the device tests (ideally move them into a new plan called Device or sth) in spare time 🚀

If you encounter sth flaky, maybe it's worth to create a 3rd suite called Flaky for investigation.

Let's make it 🟢 then try to parallelize.

@pblazej pblazej changed the title CI Test Plan CI: Test Plan Oct 2, 2025
@pblazej pblazej requested a review from hiroshihorie October 2, 2025 10:14
@pblazej pblazej force-pushed the blaze/tests-ci branch 3 times, most recently from 519ea78 to 2d34260 Compare October 3, 2025 08:39
@pblazej
Copy link
Contributor Author

pblazej commented Oct 3, 2025

Here's the status quo:

Screenshot 2025-10-03 at 10 25 24 AM

My plan is to move them to a separate target/filter out.

Unfortunately, test plans vs xcodebuild vs spm is a pain, no silver bullet here.
Moving to a separate test target is also an option (LiveKitTestsDevice etc.) but we need to extract test utils first.

@pblazej pblazej force-pushed the blaze/tests-ci branch 4 times, most recently from f59126b to 680815f Compare October 6, 2025 08:20
@pblazej pblazej changed the title CI: Test Plan CI: Test schemes Oct 6, 2025
@pblazej pblazej force-pushed the blaze/tests-ci branch 2 times, most recently from 902a8e7 to 214f522 Compare October 6, 2025 11:06
@pblazej
Copy link
Contributor Author

pblazej commented Oct 6, 2025

@hiroshihorie what I ended up doing:

  • extracted common test utils into Tests/LiveKitTestSupport target
  • split tests into 3 targets (Core/Audio (things that makes more sense on device)/ObjC)
  • removed the default test plan - for SPM it does more harm than good
  • enabled ALL Core + ObjC tests on CI
    • retry 2x
    • timeout total 10m
    • parallel (about 1.5x of speed up)

Let's see how flaky it is 🤞 Marking that as ready for review.

BTW I haven't touched the LKTestHost at all, if you've got any ideas for simplifying on-device testing, just let me know or push it in a separate PR.

@pblazej pblazej marked this pull request as ready for review October 6, 2025 12:05
@pblazej
Copy link
Contributor Author

pblazej commented Oct 7, 2025

Added tsan/asan variants to the matrix

@pblazej pblazej merged commit 4124947 into main Oct 7, 2025
39 of 47 checks passed
@pblazej pblazej deleted the blaze/tests-ci branch October 7, 2025 07:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants