Prospective Work, Chronologically

Here we outline the work that may be required if we choose to not terminate Nixpkgs CUDA and the Team early. The #Index section sorts these projects into a hierarchy by category.

Upkeep

For Nixpkgs right now, the least important, the least useful, and the hardest, yet also the most easily aligned with the “shared pool” funding model, CUDA-related work is the regular upkeep:

• carrying out timely updates with the necessary debugging and patching,
• reacting to regressions detected by the CI or reported in the forge,
• fixing mass-regressions post-staging,
• adding new software.

This work converts directly into a margin for the commercial users, but does nothing to stop the ongoing harm to Nixpkgs. This work is also the hardest to be carrying out because it is heavily compute-bound. This is the reason we should not concern ourselves with Upkeep until we’ve made substantial progress with the…

Hardware

The most critical, pressing issue for the Nixpkgs CUDA Team is hardware. All of the Upkeep and Refactoring work involves frequent “mass rebuilds”, but also requires low latency due to its interactive nature. The meaning of a “mass rebuild” in case of CUDA maintenance ranges from rebuilding an order of 1000 to 2000 packages which specifically have parts of the CUDA Toolkit as a transitive dependency, to potentially rebootstrapping a big subset (tens of thousands of derivations[83]) of Nixpkgs when the generic setup hooks or the stdenv need modification. About once every 5 to 7 days[81] or once in 2 to 3 weeks[82], respectively, the staging-nixos and staging-next branches are merged into master, at which point all of these 2000 affected packages are built and tested with CUDA support for the first time in the cycle after a low-level dependency like glibc or a compiler was modified, resulting in a spawn of CUDA-specific regressions[85][84]. Addressing these regressions early depends on building staging and staging-next in advance and regularly paying a fraction of the bootstrap cost.

The Team has been maintaining a CI for Nixpkgs’ CUDA since early 2022[86][87][88] and issuing calls for sponsorship[89][90][91][92]. The post-merge CI and the manual pre-merge nixpkgs-review rounds were carried out using a combination of primarily personal and university, but sometimes also employers’ hardware. The Team has been privately trying to begin transitioning from rented to owned and co-located compute, was ready and urgently needed to start acquiring hardware since the beginning of 2025.

Establish Metrics

We possess historical data both for “CNO”¹ and for CUDA CI: builders and build times, closure sizes, dependency graphs, build logs, versioning information, commit and merge times.[19][135] This data can be used to infer the statistics we allude to in this text, such as, but not limited to, “time-to-fix” and “time-to-update”, and their reference values.

Refactoring: The Big Three

Once the flight of maintainers and the shortage of CPU compute have been addressed, we must resolve the long-standing issues in the packages with some of the worst time-to-fix and time-to-update statistics: cf. PyTorch: git-submodules, and “Tensorflow, JAX, and XLA” in the Index. TBD: onnxruntime section. This work ultimately amounts to adding Dependency Injection support in these projects.

Prospective Work: Index

• Upkeep
• Infrastructure
  • Hardware
  • Retrospective CI: Builds
  • Retrospective CI: GPU Runtime Tests
  • Pre-Merge CI
  • Wiring Ephemeral Builders to Hydra
  • Stable Channels
• Refactoring
  • Late Binding for python3XPackages
  • PyTorch: Incremental Builds, Dependency Injection
  • Tensorflow, JAX, and XLA

Upkeep

Cf. Prospective Work, Chronologically: Upkeep.

Infrastructure

Hardware Goals

• Present: 172 rented cores, 1x rented RTX 4000, which allow rebuilding a single-variant (default cudaCapabilities, x86_64-linux) CUDA jobset (2e3 drv.) post staging-next in less than 24 hours.[84]
• CPU cores for maintainers’ remote builders and interactive jobs, crucial for refactoring, debugging, reviews, and upkeep.
• 5-10x x86_64 cores: per-capability builds. Reduce the “time until first test results”, skip fatbin-related errors (i.e. reveal “useful” errors earlier), allow for slimmer closures and images.
• Ampere cores: SBSA and Jetson builds.
• Samples of consumer-grade GPUs per architecture for observing hardware-specific regressions.
• Samples of Jetson devices per model for observing hardware-specific regressions.
• EC2 GPU instance for cloud-specific regressions.
• DGX in AWS instance for cloud-specific regressions.
• NVIDIA in Azure

Retrospective CI: Builds

Cf. proofs of the concept[93][86]

Retrospective CI: GPU Runtime Tests

Cf. proofs of the concept[94][87][95] and context[96].

Pre-Merge CI

Cf. proofs of the concept[76][77]. One way in which Nixpkgs ensures stability is by testing packages affected by pull requests prior to their merging into base branches (master, release branches). This way the committer may make an informed decision to preemptively fix a regression, delegate the decision to another expert, or deliberately break another person’s code. CUDA-dependent packages are excluded from being tested on the official infrastructure due to licensing and funding issues, and so are excluded from this workflow.

Rolling Release (Stable Channels)

Cf. proofs of the concept[78][79][80] and context[97]. Changes from master and staging do not reach nixpkgs-unstable and the release branches until the HEAD passes a set of channel-blocking tests. Due to licensing and funding issues, CUDA is excluded excluded from this workflow.

ccacheStdenv jobsets

May offer up to an order of magnitude shorter time-to-first-error at the cost of a higher false-positive rate. A binary cache for the ccacheStdenv-overlaid Nixpkgs could also greatly speed up interactive work, including the manual nixpkgs-review runs.

Hydra Maintainability & Bottlenecks

• VM Isolation
• Key Rotation
• snix-narbridge
• Side-step the Nix substitution protocol using a hierarchical snix-store
• Assess Hydra’s frequency limits for YorikSar/hydra-github-app
• Dynamic adjustment of concurrency limits: “multiple small derivations” v. “several big derivations” modes. Ack. Gaetan
• Hydra observability and the closely related Scanning of the Archives (&Outputs).

Wiring Ephemeral Builders to Hydra

Cloud Spot Instances would be needed to dynamically scale the available compute, when jobs are distributed sparsely in time. One particular scenario that matches these expectations is the testing of datacenter-grade hardware, which it may be expensive to rent and cost-inefficient to own.

Integration Testing: CUDA in SLURM

Integration Testing: CUDA in Singularity (Apptainer)

Cf. recurring regressions[98][99][100]

Integration Testing: CUDA in K3S, containerd, &c.

Cf. context[101][102][103].

Integration Testing: CUDA in AWS

Integration Testing: CUDA in Azure

Integration Testing: Nixpkgs CUDA on Jetsons

Integration Testing: Nixpkgs CUDA on Non-NixOS

Refactoring

Nixpkgs CUDA in 2026 comes with a record of five years of continuously merging technical debt after a half-decade of dormant existence, ignored by most maintainers and not observed by any automated testing. In the same vein, Nixpkgs Python is, for perfectly justfiable historical reasons, designed to curtail and prevent everything that upstream Python Packaging is designed to do, and that the Python Packaging has finally began to get a grip on in the recent (post 2020) years. Nixpkgs’ stdenv, cc-wrapper, “platforms”, and “splicing” are designed to model the simplistic world of CPU-only hardware. Nixpkgs’ bias for “early binding”, the cause of the “rebuilding the world” problem, emerged in the context where the practical feasibility of “correct” linking could be well put to question. Now that we’ve seen that correctness and determinism are indeed mostly attainable, we might have to reconsider the bulk of our past decisions as we begin to investigate whether efficient correctness and determinism are possible.

Late Binding for python3XPackages

Cf. Fridh’s[104] and adisbladis’[105] works. We expect this to reduce the size and frequency of mass rebuilds by an order of magnitude, eliminate certain conflicts with upstream (venv, poetry, uv when used impurely) packaging tooling, edge cases of PYTHONPATH leakage or downstream scripts picking up the unwrapped python interpreter, sidestep most instances of conflicting major versions of python packages (numpy1-numpy2, pydantic-pydantic2, potentially protobuf, et c.) ending up in the same transitive closure. Certain ad hoc workarounds related to reliance on nix-supported/propagated-build-inputs may be finally reverted.[106]

Migrating python3XPackages to the late binding model is a complex coordination and communication problem, due to inherent difference in the interfaces.[105]

Separate checkPhases

In the same vein as the late binding challenge, it has been a long-standing issue that checkPhases be transparently separated into their own derivations for better memoization.

Torch: git-submodules

Cf. context[107]. There have been several attempts to remove fetchModules = true from Nixpkgs’ PyTorch expressions[38]. These have drowned in the amount of work that is requried not only on the Nixpkgs side, but also upstream (e.g. abuse of add_subdirectory). A similar effort[107] has been carried out by the Guix HPC community, but they may lag behind with updates or lack CUDA support. In practice, there is likely not a single organization in the world that is actually capable of bootstrapping PyTorch from clean slate (certainly not the upstream).

Litmus tests:

• Incremental download: submodules must not be lumped into a single FOD.
• Incremental build: external dependencies (including those pinned as submodules) must be built in separate derivations, with the results “injected” into a smaller PyTorch derivation through using standard CMake tools (find_package()).
• PyTorch must not depend on the exact pinned dependencies.

Tensorflow, JAX, and XLA

Tensorflow, JAX, and OpenXLA are built using Bazel. Bazel is a build tool that implements its own sandboxing, but provides only a very limited support for dependency injection[5]; Bazel enforces only a fairly lax hermiticity policy, allowing to run impure builds and fetch unpinned dependencies[4]; Bazel’s own maintainers might unofficially suggest that Google also deviates from Bazel’s best practices, further aggravating the issue; in the sum this leads to reproducibility issues downstream[35], inflates the price of a full-source bootstrap, increases maintenance costs for all distributions, including Nixpkgs.

Poor inspect-ability, prior to bzlmod, prevents the generation of stable Fixed-Output Derivations[6][7][8]. Similarly to PyTorch[107][38], lack of Dependency Injection support upstream means that we cannot build these projects’ common dependencies (in particular, OpenXLA[39][74][75]) in isolation, solving one problem at a time, but are forced to address e.g. the OpenXLA build failures at least twice, within larger and more complex derivations, spending double build and debug time. As a result, libtensorflow has not been built from source since December of 2024 (written at: Thu Mar 5 04:44:55 AM EET 2026).

Further, buildBazelPackage’s “single ${name}-deps.tar.gz FOD²” approach, also abused by fetchCargoTarball and fetchYarnDeps, is especially prone to hash invalidation. Pinning only a single hash for a large BLOB, it prevents effective caching of intermediate or partial results. While certain packages are allowed to achieve better granularity by committing a static lock file to Nixpkgs (cf. e.g. importCargoLock), this approach does not scale, nor is necessarily feasible with Bazel or bzlmod.

On the more fundamental level these issues may be related to the limited extent of support for “Dynamism”[40] in Nix. There are two main contending implementations of dynamic build graphs: “dynamic derivations”[41], which make the ATerm “drvlang” monadic, and “import from derivation” (IFD), which makes Nixlang evaluation monadic. The “inspectability issues” in Bazel perceived by Nixpkgs maintainers could be explained as a discrepancy between Bazel’s highly granular build graphs and Nixpkgs’ current status quo in keeping dependency graphs static.

Tensorflow: CMake build

In tensorflow/53647#issuecomment-1063560009, Tensorflow maintainers have indicated they might be open to merging support for alternative build systems, in particular CMake. As with PyTorch, it would be of great value to Nixpkgs and to other distributions to implement a CMake (Meson) build for Tensorflow, and to advocate for Upstream to accept it.

Scanning The Archives

Cf. cuda-redist-find-features and sidx for POC. While the implementation diverged from the goals very early on, the origin of cuda-redist-find-features was our assessment that cudaPackages’ setup hooks should not distinguish between “cudatoolkit run-files” and the then-novel “binary archives” in the first place. Instead, we should have our own catalogue of known “components”, which we can identify in arbitrary input archives, and which we assign to specific outputs before any Nix evaluation or builds ever take place.

Complete _cuda-cudaPackages separation

The motivation for Scanning The Archives is not just observability, but the logical conclusion of the push for “stable namesets, eradication of throws, prevention of infinite recursion”.[69][70][72][109][110][111][113][71]. This was the reason for the temporary introduction[70] and for subsequent removal[69] removal of the dependency on evalModules.

Extra

The following items have a more subtle impact on Nixpkgs CUDA and their choice may be seen as more arbitrary. We’ll decide where they belong later.