(DockerHub) Proxy Cache

Infrastructure
1

Making heavy use of containers in CI quickly leads to rate-limit issues, especially for DockerHub after they have tightened their rate-limits throughout this year.

A gentle way to get around these is to use a proxy cache. This means a central instance is pulling images from DH and stores them in their registry. Users then pull from the “proxy” registry and get served the image from there, while still being able to reference DockerHub images.

We have experience running a proxy cache through a self-hosted Harbor registry for some time. Providing a central one to all CodeFloe users would make a lot of tasks easier.

The idea would be to self-host a dedicated Harbor registry for CodeFloe under cr.codefloe.com and restrict it’s use to CF CI servers. This prevents abuse from arbitrary users. Otherwise there is a likelihood that the proxy cache gets abused and we pay for all the traffic costs.

Instead of writing alpine:3.23, users would then need to write an image declaration as follows: cr.codefloe.com/docker.io/library/alpine:3.23.

Proxy caches for any registry are possible, being it quay.io, ghcr.io or others.

Please share your thoughts about this idea!

2 Likes
2026-01-10: Exclusive container registry proxy caches for CI
2

That is great news!

I have just tried it, but it fails. Am I missing something?

Error response from daemon: unauthorized: failed to connect to `host=harbor-pgbouncer user=harbor database=harbordb`: server error (FATAL: server login has been failing, cached error: connect failed (server_login_retry) (SQLSTATE 08P01)): failed to connect to `host=harbor-pgbouncer user=harbor database=harbordb`: server error (FATAL: server login has been failing, cached error: connect failed (server_login_retry) (SQLSTATE 08P01))

Crow CI

when:

  - event: manual

  - event: push

    branch: ${CI_REPO_DEFAULT_BRANCH}

 

steps:

  - name: Build pages (nix)

    image: cr.codefloe.com/docker.io/spectralcockpit/nix-datascience-alpine:4.5.1-alpine-3.22

    pull: true

    commands:

      - cp ./nix_envs/r-pages/default.nix /app/default.nix

      - nix-shell ./nix_envs/r-pages/default.nix --run 'Rscript -e "devtools::install(); altdoc::render_docs(tool = \"mkdocs\")"'

      - sed -i "s|<pre><code class='language-R'>|\n\`\`\`r\n|g" docs/man/*.md

      - sed -i "s|</code></pre>|\`\`\`\n|g" docs/man/*.md

      - mv docs/ /tmp/public

      - mv .git /tmp/.git

      - mv statichost.yaml /tmp/statichost.yaml

      - rm -rf ./* .[^.]* ..?*

      - mv /tmp/public/* ./ && mv /tmp/.git ./ && mv /tmp/statichost.yaml .

 

  - name: Push pages

    image: reg.devxy.io/docker.io/appleboy/drone-git-push:1.2.1

    settings:

      remote: ${CI_REPO_CLONE_SSH_URL}

      branch: pages

      force: true

      commit: true

      commit_message: "ci(docs): updated pages ${CI_COMMIT_SHA}"

      author_name: spectral-cockpit-bot

      author_email: bot@spectral-cockpit.com

      ssh_key:

        from_secret: spectralcockpit_bot_ssh_key
3

Temporary service issue with the registry and it’s DB. Try again, it should work now.

1 Like
4

Thank you, yes I confirm it works now :sunny:

1 Like