feat: Add s390x architecture support via cross-compilation#5300
Draft
Lan Yao (lyao-77) wants to merge 2 commits intomasterfrom
Draft
feat: Add s390x architecture support via cross-compilation#5300Lan Yao (lyao-77) wants to merge 2 commits intomasterfrom
Lan Yao (lyao-77) wants to merge 2 commits intomasterfrom
Conversation
This commit adds support for building librdkafka for the s390x (IBM Z) architecture using cross-compilation on AMD64 hosts, avoiding the performance and stability issues associated with QEMU emulation. ## Changes ### New Files **packaging/tools/Dockerfile.s390x** - Multi-stage Dockerfile using BuildKit cross-compilation - Uses tonistiigi/xx:1.5.0 toolkit for cross-compilation helpers - Builds s390x binaries natively on AMD64 platform - Avoids QEMU emulation during compilation (only used for verification) - Includes all dependencies: SSL, SASL, ZSTD, LZ4 **packaging/tools/build-s390x-cross.sh** - Build script for s390x cross-compilation - Wraps docker buildx with appropriate flags - Produces tar.gz artifacts compatible with existing build system - Includes SHA256 verification ### Modified Files **.semaphore/semaphore.yml** - Added "Linux s390x: release artifact cross-compilation builds" block - Runs on AMD64 agents with QEMU and buildx setup - Produces s390x artifacts with naming convention matching AMD64/ARM64 - Added s390x dependency to "Packaging" block ## Technical Approach ### Cross-Compilation Strategy - Build host: AMD64 (native) - Target architecture: s390x - Cross-compiler: s390x-linux-gnu-gcc/g++ - Configuration: `./configure --host=s390x-linux-gnu` ### Why Cross-Compilation vs QEMU? **QEMU Emulation Issues:** - 5-10x slower build times - Compiler instability (potential segfaults) - Complex dependency issues **Cross-Compilation Benefits:** - Fast builds (native AMD64 performance) - Stable and reliable - Well-supported by librdkafka's autotools build system ## Artifact Naming Convention Following existing pattern: ``` p-librdkafka__plat-linux__dist-ubuntu__arch-s390x__lnk-std__extra-gssapi p-librdkafka__plat-linux__dist-ubuntu__arch-s390x__lnk-all ``` ## Testing Build can be tested locally with: ```bash packaging/tools/build-s390x-cross.sh artifacts/librdkafka-s390x.tar.gz ``` Verify architecture: ```bash tar xzf artifacts/librdkafka-s390x.tar.gz file usr/local/lib/librdkafka.so.1 # Expected: ELF 64-bit MSB shared object, IBM S/390 ``` ## Future Improvements - Add --disable-gssapi variant (currently builds with gssapi) - Consider alpine musl variant for s390x - Add native s390x agent support if Semaphore provides it Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
|
🎉 All Contributor License Agreements have been signed. Ready to merge. |
There was a problem hiding this comment.
Pull request overview
This PR adds support for building librdkafka for the s390x (IBM Z) architecture using cross-compilation on AMD64 hosts, avoiding performance issues with QEMU emulation.
Changes:
- Introduces Docker-based cross-compilation tooling for s390x builds using BuildKit and the tonistiigi/xx toolkit
- Adds Semaphore CI pipeline configuration to produce s390x release artifacts on AMD64 agents
- Creates build artifacts compatible with existing naming conventions for other architectures
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| packaging/tools/Dockerfile.s390x | Multi-stage Dockerfile implementing cross-compilation for s390x using BuildKit and cross-compilation toolchain |
| packaging/tools/build-s390x-cross.sh | Build script wrapper for Docker buildx that produces tar.gz artifacts with SHA256 verification |
| .semaphore/semaphore.yml | CI pipeline configuration adding s390x cross-compilation build jobs and packaging dependency |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| - name: 'Build: cross-compiled glibc' | ||
| env_vars: | ||
| - name: ARTIFACT_KEY | ||
| value: p-librdkafka__plat-linux__dist-ubuntu__arch-s390x__lnk-all |
There was a problem hiding this comment.
The artifact key 'p-librdkafka__plat-linux__dist-ubuntu__arch-s390x__lnk-all' suggests this build includes all linkage types, but the comment indicates it's building with gssapi, which is inconsistent with the naming. The build should either actually exclude gssapi (to match the 'lnk-all' naming convention) or the artifact key should be renamed to clarify that gssapi is included.
Suggested change
| value: p-librdkafka__plat-linux__dist-ubuntu__arch-s390x__lnk-all | |
| value: p-librdkafka__plat-linux__dist-ubuntu__arch-s390x__lnk-all__extra-gssapi |
## Changes ### Dockerfile.s390x - Added libcurl4-openssl-dev to cross-compilation dependencies - Set CC, CXX, AR, RANLIB explicitly in configure command - Switched from --install-deps to using system packages - Fixed PKG_CONFIG_PATH for cross-compilation ### semaphore.yml - **TEMPORARY:** Enable s390x build on feat/s390x-cross-compilation branch for testing - Simplified to single test job (will restore both variants after successful test) - Added TODO comments to remind removal of test conditions ## Testing This commit allows the s390x build to run in Semaphore CI for validation. After successful build, will: 1. Remove branch condition (keep only tag condition) 2. Restore both build variants (with/without gssapi) 3. Ready for production use Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds support for building librdkafka for the s390x (IBM Z) architecture using cross-compilation on AMD64 hosts, avoiding the performance and stability issues associated with QEMU emulation.
Motivation
Enterprise customers operating IBM LinuxOne/Z infrastructure require librdkafka artifacts for s390x architecture. Building under QEMU emulation is too slow (5-10x) and unstable. Cross-compilation provides fast, reliable builds on existing AMD64 CI infrastructure.
Changes
New Files
packaging/tools/Dockerfile.s390x
packaging/tools/build-s390x-cross.sh
Modified Files
.semaphore/semaphore.yml
p-librdkafka__plat-linux__dist-ubuntu__arch-s390x__lnk-std__extra-gssapip-librdkafka__plat-linux__dist-ubuntu__arch-s390x__lnk-allTechnical Approach
Cross-Compilation Strategy
./configure --host=s390x-linux-gnuWhy Cross-Compilation vs QEMU?
Testing
Local Testing
CI Testing
Future Improvements
Related Work
Co-Authored-By: Claude Sonnet 4.5 [email protected]