Bazel: rules_multirun Python dependency prevents local `bazel run` with remote execution #869

New issue

Open

opened 2026-03-28 04:29:24 +00:00 by mfreeman451 · 0 comments

mfreeman451 commented

2026-03-28 04:29:24 +00:00

Owner

Imported from GitHub.

Original GitHub issue: #2517
Original author: @mfreeman451
Original URL: https://github.com/carverauto/serviceradar/issues/2517
Original created: 2026-01-26T05:32:53Z

Problem

When using remote execution (--config=remote) from macOS to build Linux Docker images, bazel run //docker/images:push_all fails with:

OSError: [Errno 8] Exec format error: '.../rules_python++python+python_3_11_x86_64-unknown-linux-gnu/bin/python3'

The same issue affects individual oci_push targets (they embed Linux jq binary).

Root Cause

rules_multirun is implemented in Python. When using remote execution:

Bazel builds targets for the execution platform (Linux x86_64)
Host-transitioned targets (like rules_multirun) embed Linux Python in runfiles
bazel run executes locally on macOS
Linux binaries cannot execute on macOS → "Exec format error"

This is a known limitation of Bazel's platform model when host ≠ execution platform.

Upstream References

Issue: https://github.com/keith/rules_multirun/issues/43 (open since 2023)
Go Port PR: https://github.com/keith/rules_multirun/pull/57 (stalled since June 2024)
Aspect blog on prebuilt binaries: https://blog.aspect.build/releasing-bazel-rulesets-rust

The maintainer is "generally supportive" of switching to Go but PR #57 has been waiting for review. The PR also has a bug where go_sdk extension isn't properly guarded as dev_dependency.

Current Workarounds

Push from CI only - GitHub Actions runs on Linux where host = execution platform
scripts/push_all_images.sh - Wrapper script that explains the limitation
Build works fine - make build / bazel build //... works with remote execution

What Still Works

Action	Works on macOS?	Notes
`make build`	✅	Builds all images via RBE
`bazel build //docker/images:...`	✅	All images built correctly
`bazel run //docker/images:push_all`	❌	Python exec format error
`bazel run` from Linux CI	✅	Host matches execution platform

Potential Solutions

Wait for upstream - Monitor PR #57 for progress
Fork and fix PR #57 - Fix the dev_dependency bug, maintain our own fork
Replace rules_multirun - Simple shell script that calls each push target
Contribute upstream - Help get PR #57 merged

Files with Documentation

MODULE.bazel - Comment explaining the issue
.bazelrc - Note about bazel run limitation with remote execution
scripts/push_all_images.sh - Workaround script for CI
Makefile - Updated push_all target

Labels

bazel
toolchain
wontfix (for now)

Imported from GitHub. Original GitHub issue: #2517 Original author: @mfreeman451 Original URL: https://github.com/carverauto/serviceradar/issues/2517 Original created: 2026-01-26T05:32:53Z --- ## Problem When using remote execution (`--config=remote`) from macOS to build Linux Docker images, `bazel run //docker/images:push_all` fails with: ``` OSError: [Errno 8] Exec format error: '.../rules_python++python+python_3_11_x86_64-unknown-linux-gnu/bin/python3' ``` The same issue affects individual `oci_push` targets (they embed Linux `jq` binary). ## Root Cause `rules_multirun` is implemented in Python. When using remote execution: 1. Bazel builds targets for the execution platform (Linux x86_64) 2. Host-transitioned targets (like `rules_multirun`) embed Linux Python in runfiles 3. `bazel run` executes locally on macOS 4. Linux binaries cannot execute on macOS → "Exec format error" This is a known limitation of Bazel's platform model when host ≠ execution platform. ## Upstream References - **Issue**: https://github.com/keith/rules_multirun/issues/43 (open since 2023) - **Go Port PR**: https://github.com/keith/rules_multirun/pull/57 (stalled since June 2024) - **Aspect blog on prebuilt binaries**: https://blog.aspect.build/releasing-bazel-rulesets-rust The maintainer is "generally supportive" of switching to Go but PR #57 has been waiting for review. The PR also has a bug where `go_sdk` extension isn't properly guarded as dev_dependency. ## Current Workarounds 1. **Push from CI only** - GitHub Actions runs on Linux where host = execution platform 2. **`scripts/push_all_images.sh`** - Wrapper script that explains the limitation 3. **Build works fine** - `make build` / `bazel build //...` works with remote execution ## What Still Works | Action | Works on macOS? | Notes | |--------|-----------------|-------| | `make build` | ✅ | Builds all images via RBE | | `bazel build //docker/images:...` | ✅ | All images built correctly | | `bazel run //docker/images:push_all` | ❌ | Python exec format error | | `bazel run` from Linux CI | ✅ | Host matches execution platform | ## Potential Solutions 1. **Wait for upstream** - Monitor PR #57 for progress 2. **Fork and fix PR #57** - Fix the dev_dependency bug, maintain our own fork 3. **Replace rules_multirun** - Simple shell script that calls each push target 4. **Contribute upstream** - Help get PR #57 merged ## Files with Documentation - `MODULE.bazel` - Comment explaining the issue - `.bazelrc` - Note about `bazel run` limitation with remote execution - `scripts/push_all_images.sh` - Workaround script for CI - `Makefile` - Updated `push_all` target ## Labels - `bazel` - `toolchain` - `wontfix` (for now)