Per-project binary caches (or alternatives)

[dhess]

We're preparing to make a few open source releases, and we'd like to provide public Cachix caches for these projects. However, we have multiple other projects that will remain private, and we'd like to push derivations to either a private S3 or private Cachix cache for these projects. As far as I can tell, Hercules CI doesn't easily support this scenario. At the moment, I believe we'd need to create one account/org for the private builds, and one for the public builds, and then partition our builders by account/org.

One way to nicely support this would be to configure binary cache outputs on a per-project level; e.g., when you add a GitHub repo to Hercules CI, you can also specify the binary cache to use for builds for that project.

Another way would be per-project agents. In this scenario you'd assign different agents to different projects, and the user would need to ensure that each agent used the correct binary cache. This seems less ideal, not only because of the configuration headaches, but also because I think it wouldn't be possible to use the same host to build both private and public projects, unless you also added support for multiple agents per host.

I don't know much about Hercules CI Effects, as I haven't used Hercules CI since these were added. Perhaps what I want to do is possible via effects?

[MatthewCroughan]

The lack of multi-project support in a single agent, or at least the ability to run multiple agents on the same machine has become a bit of a blocker for me, as running the agent in a container is not a real solution, since it is buggy in quite a few ways, namely:

• https://github.com/hercules-ci/hercules-ci-effects/issues/45

• Running agent inside NixOS container hercules-ci-agent#345

  • I have to assign static user IDs for files accessed by the guest, like secrets.
  • I have to duplicate a lot of Nix code, like making sure nixUnstable is enabled inside each container.
  • Container networking on boot is a bit weird, sometimes DNS isn't there, probably something to do with my host networking, either way it's not ideal.

[dhess]

Is it safe to say that this has been addressed by hercules-ci/hercules-ci-agent#361 (comment) ?

(If so, I suppose only for NixOS machines and not Mac builders. Any plans to make this feature available on macOS?)

[roberth]

Is it safe to say that this has been addressed by hercules-ci/hercules-ci-agent#361 (comment) ?

Only if the caches correspond exactly to github organizations.

(If so, I suppose only for NixOS machines and not Mac builders. Any plans to make this feature available on macOS?)

Can do so in the short term if this addresses your issue i.e. if you have a github organization per cache configuration.

[dhess]

Is it safe to say that this has been addressed by hercules-ci/hercules-ci-agent#361 (comment) ?

Only if the caches correspond exactly to github organizations.

I'm not exactly sure what you mean by this. We have one GitHub org, https://github.com/hackworthltd. That GitHub org has an associated Hercules CI account, and an associated Cachix account. The Cachix org account has 2 caches: one private, and one public. We'd like agents belonging to our Hercules CI account to push to the org's public cache when building public GitHub projects belonging to the organization, and to the org's private cache when building private projects belonging to the organization.

When I wrote the OP in Dec 2021, it was my understanding that this scenario wasn't supported. Quoting myself from the OP:

Another way would be per-project agents. In this scenario you'd assign different agents to different projects, and the user would need to ensure that each agent used the correct binary cache. This seems less ideal, not only because of the configuration headaches, but also because I think it wouldn't be possible to use the same host to build both private and public projects, unless you also added support for multiple agents per host.

I was hoping that now that it's possible to run multiple agents per (NixOS) host, it would also be possible to configure each agent to push to a particular Cachix cache. Then we could configure one agent to build public projects, and another to build private projects, and the built products would be pushed to the appropriate Cachix cache.