Skip to content

fix: add musl targets for Linux installer fallback#1013

Merged
ilblackdragon merged 3 commits intonearai:stagingfrom
rajulbhatnagar:fix/musl-installer-targets
Mar 22, 2026
Merged

fix: add musl targets for Linux installer fallback#1013
ilblackdragon merged 3 commits intonearai:stagingfrom
rajulbhatnagar:fix/musl-installer-targets

Conversation

@rajulbhatnagar
Copy link
Copy Markdown
Contributor

Summary

  • Add x86_64-unknown-linux-musl and aarch64-unknown-linux-musl to the cargo-dist target list so the installer can fall back to statically linked binaries when the system glibc is too old
  • Switch rig-core from reqwest-tls (default, pulls in OpenSSL via native-tls) to reqwest-rustls (pure Rust TLS) — this eliminates the openssl-sys transitive dependency that would break musl/static builds

Root cause

The installer requires glibc >= 2.35 (x86_64) / >= 2.39 (aarch64) because the release binaries are built on Ubuntu 22.04/24.04. Systems like Amazon Linux 2023 (glibc 2.34) get:

System glibc version (`2.34') is too old; checking alternatives
ERROR: no compatible downloads were found for your platform x86_64-unknown-linux-gnu

The cargo-dist installer already has fallback logic for musl builds, but no musl targets were configured.

Note for reviewers

After merging, the release workflow will need musl toolchains on CI runners. cargo-dist v0.30.3 handles this automatically when musl targets are present, but verify the generated CI installs musl-tools on the Linux runners.

Closes #1008

@gemini-code-assist
Copy link
Copy Markdown
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@github-actions github-actions bot added scope: dependencies Dependency updates size: XS < 10 changed lines (excluding docs) risk: medium Business logic, config, or moderate-risk modules contributor: new First-time contributor labels Mar 12, 2026
Copy link
Copy Markdown
Collaborator

@zmanian zmanian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good change overall -- the musl target additions are correct and the rig-core TLS switch is the right call. Two issues to address before merging:

1. Missing Cargo.lock update (blocking)

The diff only touches Cargo.toml but does not include an updated Cargo.lock. The rig-core feature change from default (reqwest-tls -> native-tls -> openssl-sys) to reqwest-rustls will not take effect until the lock file is regenerated. I confirmed that on the current lock file, native-tls and openssl-sys are still present, pulled in exclusively through the rig-core -> reqwest chain. Please run cargo generate-lockfile (or cargo update -p rig-core) and include the updated lock file. Without this, the musl builds will fail trying to link OpenSSL.

2. No custom runner entries for musl targets (non-blocking, worth considering)

The [workspace.metadata.dist.github-custom-runners] section has entries for all existing targets but none for the two new musl targets. cargo-dist v0.30.3 will fall back to its default Ubuntu runner, which should work since it auto-installs musl-tools, but you may want to pin specific runners for reproducibility (e.g., x86_64-unknown-linux-musl = "ubuntu-22.04" and aarch64-unknown-linux-musl = "ubuntu-24.04-arm"). This is optional but worth noting.

Everything else looks correct:

  • Target triple names are right (x86_64-unknown-linux-musl, aarch64-unknown-linux-musl)
  • The rig-core feature switch to reqwest-rustls is the clean fix -- eliminates the only path to openssl-sys in the tree
  • Existing glibc targets are untouched, no regression risk there
  • cargo-dist's installer already has the fallback logic to prefer glibc and fall back to musl, so no installer script changes are needed

@rajulbhatnagar rajulbhatnagar marked this pull request as draft March 13, 2026 00:05
@github-actions github-actions bot added size: M 50-199 changed lines and removed size: XS < 10 changed lines (excluding docs) labels Mar 13, 2026
@rajulbhatnagar rajulbhatnagar force-pushed the fix/musl-installer-targets branch from 0b28b3f to 54c41c0 Compare March 13, 2026 00:09
@rajulbhatnagar rajulbhatnagar marked this pull request as ready for review March 13, 2026 00:11
@gemini-code-assist
Copy link
Copy Markdown
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

Copy link
Copy Markdown
Collaborator

@zmanian zmanian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The approach is correct -- adding musl targets for glibc fallback and switching rig-core to rustls to eliminate the OpenSSL dependency that would break static builds.

However, there are issues:

1. CI is not running full checks (blocking)

Only classify and scope CI jobs ran. The main build, clippy, and test jobs did not execute. This is a significant concern because:

  • The Cargo.lock changes are substantial (clap, anstream version bumps, OpenSSL removal)
  • We need to verify that removing native-tls / openssl-sys doesn't break anything downstream
  • The musl target configuration needs validation

Please rebase and ensure full CI runs before this can be reviewed properly.

2. Verify cargo-dist musl toolchain support

The PR mentions cargo-dist v0.30.3 handles musl toolchains automatically, but the CI runners are set to ubuntu-22.04 and ubuntu-24.04-arm. Confirm that musl-tools / musl-gcc are available on these runners or that cargo-dist installs them.

3. Unrelated dependency bumps in Cargo.lock

The lock file includes version bumps for clap (4.5.60 -> 4.6.0), anstream (0.6.21 -> 1.0.0), once_cell (1.21.3 -> 1.21.4) that appear unrelated to the rig-core TLS change. Were these intentional or a side effect of cargo update? Unrelated dep changes should ideally be in a separate PR to keep the diff reviewable.

4. PR description is good

The root cause analysis and reviewer note are well-written. The rig-core feature flag change (default-features = false, features = ["reqwest-rustls"]) is the right approach.

Please get full CI green and address the dep bump question, then this should be ready.

@rajulbhatnagar rajulbhatnagar marked this pull request as draft March 18, 2026 02:10
brajul added 2 commits March 18, 2026 02:10
The installer fails on systems with glibc < 2.35 (e.g. Amazon Linux
2023) because only gnu targets are built and there is no static fallback.

- Add x86_64-unknown-linux-musl and aarch64-unknown-linux-musl to the
  cargo-dist target list so the installer can fall back to statically
  linked binaries when glibc is too old.
- Switch rig-core from reqwest-tls (OpenSSL) to reqwest-rustls (pure
  Rust TLS) to avoid a system OpenSSL dependency that breaks musl builds.

Closes nearai#1008
Address review feedback:
- Regenerate Cargo.lock to reflect rig-core reqwest-rustls switch,
  removing openssl-sys and native-tls from the dependency tree
- Add github-custom-runners entries for musl targets
@rajulbhatnagar rajulbhatnagar force-pushed the fix/musl-installer-targets branch from 54c41c0 to bca8bbc Compare March 18, 2026 02:11
@rajulbhatnagar
Copy link
Copy Markdown
Contributor Author

Addressing review feedback (rebased, v3)

Changes in this update:

  • Rebased onto latest staging (2784cef)
  • Cargo.lock regenerated surgically via cargo update -p rig-core — the diff now only removes the 9 OpenSSL/native-tls transitive packages (openssl-sys, openssl, openssl-macros, native-tls, tokio-native-tls, hyper-tls, foreign-types, foreign-types-shared, vcpkg). No unrelated dependency version bumps.

Feedback items:

# Feedback Status
1 CI not running full checks The test.yml workflow only triggers on PRs targeting main (branches: [main]). Since this PR targets staging (the repo default branch), only classify, scope, and code_style run. Ran the full CI test matrix locally — results below.
2 Verify musl toolchain support The current release.yml is autogenerated by cargo-dist and does not yet include musl build steps — those will be generated when cargo dist generate is next run after these targets are in Cargo.toml. cargo-dist v0.30.3 auto-installs musl-tools when it detects musl targets in the build plan.
3 Unrelated dep bumps in Cargo.lock Fixed — used cargo update -p rig-core instead of cargo generate-lockfile. Diff now shows only the OpenSSL removal, zero unrelated version bumps.

Local CI results (matches test.yml + code_style.yml matrix):

Check Result
cargo fmt --all -- --check pass
Clippy (all-features) pass (0 warnings)
Clippy (default) pass (0 warnings)
Clippy (libsql-only) pass (0 warnings)
Tests (all-features): --no-default-features --features postgres,libsql,html-to-markdown,bedrock,import 3239 passed, 3 ignored
Tests (default) 3197 passed, 3 ignored
Tests (libsql-only): --no-default-features --features libsql 3188 passed, 2 ignored

openssl-sys / native-tls occurrence count in Cargo.lock: 0 (confirmed removed)

@rajulbhatnagar rajulbhatnagar marked this pull request as ready for review March 18, 2026 02:58
@rajulbhatnagar rajulbhatnagar requested a review from zmanian March 18, 2026 02:59
Copy link
Copy Markdown
Collaborator

@zmanian zmanian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clean infrastructure fix. Adding musl targets with rustls (pure Rust TLS) eliminates the OpenSSL cross-compilation headaches for statically-linked Linux binaries.

Strengths:

  • Removes openssl, native-tls, hyper-tls dependencies (pure Rust TLS stack now)
  • Adds both x86_64 and aarch64 musl targets as fallback installers
  • All CI checks pass

Post-merge suggestion: Verify in CI logs that musl-tools gets installed on Linux runners and both musl builds succeed. Also validate no regressions from the socket2 version change (0.6.3 -> 0.5.10).

Approving.

@ilblackdragon ilblackdragon added the skip-regression-check Bypass regression test CI gate (tests exist but not in tests/ dir) label Mar 21, 2026
@github-actions github-actions bot added contributor: regular 2-5 merged PRs and removed contributor: new First-time contributor labels Mar 21, 2026
Copy link
Copy Markdown
Member

@ilblackdragon ilblackdragon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

Clean, well-motivated fix for the glibc version mismatch on systems like Amazon Linux 2023.

Changes are sound:

  • musl targets added with correct CI runner mappings
  • rig-core switched from native-tls to reqwest-rustls, cleanly removing openssl-sys and all transitive OpenSSL dependencies
  • webpki-roots correctly added to provide bundled Mozilla root CAs for static builds

One note for release notes: The switch from native-tls to rustls means TLS verification now uses a bundled Mozilla root CA set instead of the OS certificate store. Environments with custom/corporate CA certificates not in Mozilla's bundle will fail TLS verification. Worth calling out for users behind corporate proxies with MITM certs.

Minor: The Cargo.lock includes some transitive version bumps (windows-sys, socket2, getrandom) — these are from the rig-core feature switch resolution, not manually introduced.

Verify the first release build after merge installs musl-tools on the CI runners as expected.

LGTM.

@ilblackdragon ilblackdragon merged commit 0e5837b into nearai:staging Mar 22, 2026
14 checks passed
bkutasi pushed a commit to bkutasi/ironclaw that referenced this pull request Mar 28, 2026
…-targets

fix: add musl targets for Linux installer fallback
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

contributor: regular 2-5 merged PRs risk: medium Business logic, config, or moderate-risk modules scope: dependencies Dependency updates size: M 50-199 changed lines skip-regression-check Bypass regression test CI gate (tests exist but not in tests/ dir)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Installer fails on Linux with glibc < 2.35 — no musl/static fallback

3 participants