I’d love to see a documented example or better support for Nix builds on CrowCI. I’m waiting for NixCI to add native support for arbitrary Forgejo instances, but in the meantime, I’m using CrowCI as kind of an orchestrator for nix builds and using either NixCI or nixbuild.net for the actual build jobs.
One thing that would make this flow smoother is the ability to persist the Nix store locally in the cache so it doesn’t have unconditionally build or pull from the external builders.
If you use the local back-end nothing special has to be done at all - the nix store is persisted. If you are using a hosted runner there isn’t any caching out-of-the-box between runs other than incidental docker layer caching. There is a plugin for general cache support but you still need to bring your own storage bucket.
There is a Woodpecker plugin for attic. A hosted service for this could be really nice but I’d wonder about abuse potential.
I’m using a self-hosted runner with the local back-end. Builds are very fast this way, but honestly your nixbuild.net example looks better to me than all of these options. 0.12 per CPU-hour and 25 free hours a month. Do you find you are getting a lot of caching with it?
A hosted service from CF’s side wouldn’t be impossible but it needs care with respect to the overall rollout and configuration. It would also cost the platform quite some money long-term. Less for storage, more in regards to transaction costs. In the end, this would need user/org scoped storage buckets and an easy way to opt-in and out.
Not impossible, just not something one integrates within a few days. This is independent of nix btw, i.e. the existence of a central caching solution within the platform is surely important for CI in general. While there are many ways one can do so on the individual level already now, many just don’t do so (or don’t know how) and therefore run CI over and over again without cache, which puts quite a heavy load onto the CI runners, especially on their disks.
That looks indeed quite generous. I wonder if this tier holds under heavy use given the potential costs it might potentially incur. Sure, optimizing towards the re-use of local assets is nice and probably also essential for using nix in a performant and economic way long-term, yet
In general though, building a specialized CI solution for just one language seems like an anti-pattern to me and (one reason more) why I might not get into nix itself in the near future. It feels to me that it wants to redo everything from the ground up and be different to everything else that exists, but this is just a personal view of mine which could also be wrong.
sccache is a somewhat specific caching solution that doesn’t work for every language. Not sure if it applies to nix.
The most generic approach is likely to use the S3 plugin to cache/restore the cache, but that might also take a while.
There might be more sophisticated GitHub Actions around this that can re-use the integrated artifact caching within Forgejo Actions for this. FJ Actions might be better suited for nix jobs as of today but somebody needs to figure this out first. Crow isn’t really optimized for nix usage as of today.
You are right, this doesn’t apply here. I thought it just gave me an ~/.sccache I can do anything with, but it wraps the build steps.
Maybe a better option than using any CI cache is using an s3 store directly as a remote nix binary cache. This way only the objects needed for any particular build or step are fetched.
I’ll give this a try tonight to confirm it, but I think it could look something like this:
I think this happens because its design imposes different constraints than most systems - the awkwardness we’re dealing with here is the main reason there are things like dedicated cache and build services.
NixBuild.net and NixCI both have built-in caches already. I don’t think adding a remote cache would offer any benefit over what I already get with those providers.
I was thinking of a local Nix cache that persists across runs so that it’s available locally without the runner having to pull from remote Nix stores on build.
I know it’s maybe too expensive or impractical, but I wanted to float the idea to see if it’s viable.
The problem is you can’t safely share nix binary caches. People could poison them with binaries that claim to be one thing but are built with extra logic that steals credentials or whatever. Also you’d not want to blindly share your built binaries with others if they contain private code. I think it might be possible for a plugin to make it look like you are using a local cache, but you still need storage for it, so like you said its not better than what you are using already.
I agree this is really important. I would love to see the ability for people to use their allotted storage for cache as well as packages and repos (or a separate allocation). I think there should be a way to do this that is as economical as other storage CF is already providing. This is something I’d be willing to work on directly.
I’m not sure this definitely the case. Impossible terms limit outgoing bandwidth to storage volumes, which probably rules out this use case. But OVH also offers ~€8/TB/month on 1-AZ Standard in Frankfurt (you have to pick Frankfurt specifically to see that price), and there is “no limit” or cost per egress or API operation. 5GB per user limit would cost at most €0.04 per user monthly.
Fair point. Scaleway is even a bit cheaper than OVH here and also has 75G of free traffic and free DELETE operations.
Yes, the bandwith cap against the stored volume on Impossible is not ideal for such a use case where most of the cost will come from transactions. If we go this way, we should also likely increase the free package storage per use as the current quota doesn’t really reflect the object storage cost. It was based on what CB uses, but they store the package assets on disk, which is a completely different scenario.
Unlike the woodpecker plugin above, my version closely follows how the GitHub Action’s caching functionality works, and I found it works more reliably as a result. Of course, you’ll still need an attic server to push to!
I’m using CrowCI to build and deploy nixosConfigurations as well as building packages - the readme linked above has some simple examples to try.
Thanks! I will have a look at it. I have currently quite a bit of repos consuming nix derivations, so adding caching would make things faster and reusable
I have been following the thread, because I am currently also looking for a caching solution to address long Nix builds in my agentic tool configuration (https://codefloe.com/philipp-baumann/agentic-tooling-and-configs). i have done some work on project-based caches as a co-author of rix, an opinionated tool to use Nix for R. Now I am hitting CI limits in terms of build times, particularly for the security scanning steps, which I cannot justify to myself anymore. So I want to improve on this.
It is maybe not the most straightforward and proactive discussion to read about the security concerns, but the common takeaway is that there are positions on both private and public Nix caches.
If build systems are compromised, regardless of the underlying technology, certainly the downloaded .narinfo (plain-text metadata file that a Nix binary cache serves, including an ed25519 signature for one store path) is also compromised. Meaning download and inclusion into one’s own cache succeeds. That is by design and no surprise. With require-sigs = false, the NAR-hash check still catches transfer corruption, but since the compromised build system also writes the narinfo, nothing guarantees that the key holder’s infrastructure actually produced the artifact. Signatures provide provenance, and requiring them is the consuming client’s responsibility, be it CI/CD or directly user-facing machines.
I personally would like to set up a private cache for myself but make the build system openly accessible. I don’t expect CodeFloe to provide a community solution for it, but would of course support such efforts in the future. Also, I don’t think we need another CI/CD solution for this, as Crow CI and container-agnostic ways already provide the basics for it. Concretely I plan: self-hosted attic builder that provisions caches to S3-compatible storage, per-cache signing keys with require-sigs = true on all consumers, and server-side signing so CI runners hold push tokens but never the private key.
For the moment being, I did a preliminary, AI-assisted analysis (which is not necessarily correct in all ways), but which gives me a first starter to set up a “private” build infra for my current problem, but with openly shared configs: https://codefloe.com/philipp-baumann/agentic-tooling-and-configs/issues/286. I will continue based on the favored strategy there for the named repo.
My motivation is that I integrate build machines making the private cache, including an Apple silicon builder I control.
I am open to discussion, however would first build something and see if it fits my use. I’ll report back how it goes. Then, if anything new or useful comes out of it, this could be discussed/challenged together.
