Can't use codefloe for nixpkgs

Site Feedback
1

I am trying to build a nix package for a project that is hosted on codefloe.

Instead of getting the source code, the builder gets the page
Your browser is being checked so that you can read it without signing in

Which is obviously not ideal, and I don’t know how to work around that.

The full url used by the package is: https://codefloe.com/InkyCap/app/archive/b4eafabdb2f5611ac93efb7b776c1d0da2ce9be5.tar.gz

since nixpkgs pins revisions and/or tags.

I do not know if nixpkgs uses curl, wget or whatever to do its work.

Is there a way to work around this?

2

We put this challenge protection into effect to protect the instance from aggressive scrapers scanning arbitrary tarballs from archived paths.

There are of course legitimate use cases like yours that should ideally not be blocked by this, let alone by an interactive challenge for a non-interactive request.

We’ve now changed the approach to allow a small unauthenticated budget for non-interactive requests with the hint to use authenticated requests for automation purposes. Unfortunately, Forgejo doesn’t have a built-in logic to properly rate-limit authenticated requests to various expensive endpoints. We are currently building a generic approach to this right now.

Additionally, we ensured that CLI-based requests don’t get the browser challenge presented anymore. Can you check again if this now works on your end? Please try to use authenticated requests (i.e. supplying a token in the Authorization header) in your automation.

3

I am now able to download and build it without authentication, thank you.

Unfortunately nixpkgs can’t use any extra Authorization token or header, since it’s all public and that would mean instantly making the token public, too.

4

There must be any other way to properly authenticate nix requests then. Acting fully anonymous without a distinct UA header or similar makes the requests highly similar to any anonymous scraper.

The budget for such is deliberately low on our end and you might very likely run into rate limits then.

Let us know if there’s any way to distinguish such nixpks requests more reliably.

Hard for us to follow as we don’t know where and how you run the requests (and it sounds you don’t do so within our public CI), but the job of secret sanitization in public build environments is something that is outside of our responsibility. It does sound a bit strange though that there can’t we any secret within any request (e.g. to overcome rate-limiting) in nixpkgs requests and that everything must fit into anonymous limits of the receiving side.

5

Assuming you have your own flake where you are including this package you can use an SSH URL in the flake input section, there is an example here (yes this repo is not private, but you can clone it with any SSH key since it is public):

{
  inputs = {
    clan-core.url = "https://git.clan.lol/clan/clan-core/archive/main.tar.gz";

    my-private-pkgs = {
      type = "git";
      url  = "ssh://gitea@gitea.example.com/Luis/my-private-nix-packages.git";
    };
  };

  # ... outputs
}
6

Yes, except I was not writing a personal package.

There is a project hosted on codefloe, and I wrote the package to be included in nixpkgs so that everybody using nix can have access to the package.

After digging a bit around it seems that the requests should include the user agent:

curl/‹version› Nixpkgs/‹version›

e.g. “curl/… Nixpkgs/26.05”
It does not seem to be completely standardized and they might have Nix/26.05 , too

There is an open bug for this: Standardize User-Agent format for all fetchers in nixpkgs · Issue #513006 · NixOS/nixpkgs · GitHub

This is still not proper authentication, and I don’t think they have any intention of having that: The whole point of nixpkgs is that the package repo is the only thing you need to rebuild the packages.

Once a package has been built however it should be included in the nix caches, so you should not see too many hits.

7

A user agent that allows higher rate limits would just be adopted by the crawlers, which they’ve already done many times. But I don’t know that tarball fetches are a usage pattern that causes problems with the bots. My understanding is the main issue is they follow every rendered hyperlink, so they look at every file and commit in a repo one at a time. A rule that was more generous for tarballs probably wouldn’t open an abuse vector too wide.