:focal(smart))
rattler-buildconda-buildrattler-buildCross-compilation involves building software on one platform (the build machine) that will run on a different platform (the host machine). This is increasingly important as we see more diverse architectures in production environments—from ARM-based servers to Apple Silicon laptops.
If everything is configured correctly, cross-compilation with rattler-build is as easy as saying:
In the context of Conda packages, cross-compilation allows us to:
Build packages for ARM architectures on x86_64 CI runners
Create packages for different operating systems
Support embedded systems and specialized hardware
Reduce build times by leveraging powerful build servers. One could also build the software using emulation, but that is usually much slower than using cross-compilation which can run at native speed
The distinction between build and host environments is crucial for successful cross-compilation. This terminology has its roots in GNU Autotools, which established the convention of three distinct platforms in cross-compilation scenarios:
Build: The platform where the compilation is happening
Host: The platform where the compiled code will run
Target: The platform that the compiled code will generate output for (only relevant for compilers and similar tools)
In the Autotools world, these are specified with flags like --build, --host, and --target. For example, when cross-compiling GCC itself, you might use:
The Conda ecosystem adopted this terminology but simplified it for most use cases, focusing primarily on build and host environments since most packages aren't compilers themselves. Let's break down what each environment represents:
The build environment contains tools and dependencies that need to run during the package building process. This includes:
Compilers (GCC, Clang, Rust compiler)
Build tools (CMake, Make, Cargo)
Code generators and preprocessors
Any tool that executes during the build
The host environment contains libraries and headers that the package being built will link against. These dependencies:
Must be compatible with the target platform
Include shared libraries, static libraries, and development headers
Run exports take care that they are used by the final package at runtime (if they're shared libraries)
Here's a practical example of a rattler-build recipe that demonstrates this separation:
One of rattler-build's elegant features is the ${{ compiler() }} function. When cross-compiling, this function automatically selects the appropriate cross-compiler for your target platform.
For example, when building from linux-64 to linux-aarch64:
The ${{ compiler('c') }} function will resolve to a cross-compiler package like gcc_linux-aarch64 that:
Runs on linux-64 (your build platform)
Produces binaries for linux-aarch64 (your target platform)
This means the actual package selected might be something like gcc_linux-aarch64 which is built to run on the build platform but targets the host platform.
The magic behind Conda's cross-compilation support lies in compiler activation scripts. These scripts are part of the compiler packages and automatically set up the build environment with the correct flags and paths.
When a compiler package is installed, it includes activation scripts that:
Set environment variables like CC, CXX, CFLAGS, LDFLAGS
Configure the sysroot for cross-compilation
Set up paths to the correct headers and libraries
Define target-specific compiler flags
For example, a typical activation script might set:
These activation scripts ensure that both the build and host environments are properly configured during the build process.
You can find the sources for the activation scripts in the compiler repositories on conda-forge. And for macOS / Clang here.
One of the most innovative aspects of rattler-build is its approach to cross-platform testing. The tool provides separate build and run environments within the test section, enabling you to test cross-compiled packages even when you can't natively execute them.
This separation allows for sophisticated testing strategies:
Build-time tests: Verify that files are installed correctly, check pkg-config files, validate metadata
Runtime tests: When possible (using emulation or actual hardware), run the compiled binaries
Cross-platform validation: Use QEMU or other emulators to test ARM binaries on x86_64 systems
rattler-build can utilize QEMU for testing cross-compiled packages. The CROSSCOMPILING_EMULATOR environment variable points to the appropriate QEMU binary:
Sometimes you need different dependencies when cross-compiling:
Always distinguish build and host dependencies: Tools that run during build go in build, libraries to link against go in host
Use compiler functions: Let rattler-build handle compiler selection through ${{ compiler() }}
Test thoughtfully: Design tests that can validate your package without necessarily running target binaries
Handle environment variables: Some build systems need hints about cross-compilation:
Document platform-specific behavior: Make it clear in your recipe when certain features are only available on specific platforms
Find examples on Github: The powerful search on Github can help a lot to find good examples for cross-compilation. Just search for "CONDA_BUILD_CROSS_COMPILATION" in the conda-forge org.
Cross-compilation is a first-class citizen in the Conda ecosystem. The clear separation of build and host environments, combined with sophisticated testing capabilities and automatic compiler activation scripts, enables developers to:
Build packages for multiple architectures from a single CI/CD pipeline
Reduce infrastructure costs by centralizing builds
Ensure consistency across platforms
Test packages more thoroughly before release
We want to make cross-compilation withrattler-build extremely simple (just set --target-platform=...). Do let use know if you have feedback.
Another cool use case for cross-compilation is the emscripten-forge – where everything is cross-compiled to WebAssembly.
Chat with us on Discord if you have any questions: https://discord.gg/kKV8ZxyzY4
rattler-build --target-platform={linux-aarch64, osx-64, ...}./configure --build=x86_64-pc-linux-gnu \
--host=aarch64-linux-gnu \
--target=aarch64-linux-gnurattler-build build --target-platform linux-aarch64 recipe.yamlexport CC=$PREFIX/bin/x86_64-conda-linux-gnu-gcc
export CXX=$PREFIX/bin/x86_64-conda-linux-gnu-g++
export CFLAGS="${CFLAGS} --sysroot=$PREFIX/$HOST/sysroot"
export CMAKE_PREFIX_PATH=$PREFIX/$HOST/sysroot/usr:$PREFIXtests:
- script:
- if [[ "$CONDA_BUILD_CROSS_COMPILATION" == "1" ]]; then
$CROSSCOMPILING_EMULATOR python -c "import mymodule; mymodule.test()"
firequirements:
build:
- ${{ compiler('c') }}
- ${{ compiler('cxx') }}
- cmake
- ninja
- pkg-config
host:
- zlib
- openssl
- boost-cpp
run:
# - libzlib (automatically added as run export from `zlib`)
# - openssl (automatically added as run export from openssl)tests:
- script:
- test -f $PREFIX/lib/libmylib.so # Runs in build environment
- python -c "import mymodule" # Would run in host environment
requirements:
build:
- python
- pytest
run:
- python
- numpyrequirements:
build:
- ${{ compiler('c') }}
- ${{ compiler('cxx') }}
- if: build_platform != target_platform
then:
- qemu-user-static # For testing
- cross-python_${{ target_platform }}
host:
- python
- numpy
if [[ "${CONDA_BUILD_CROSS_COMPILATION}" == "1" ]]; then
export CMAKE_CROSSCOMPILING=ON
export CMAKE_SYSTEM_NAME=Linux
export CMAKE_SYSTEM_PROCESSOR=${target_platform##*-}
fi