On November 13 we moved codefloe.com from a single server to three Forgejo nodes that serve traffic at the same time. As far as we know, CodeFloe is the first public Forgejo-based instance to run like this: active-active, with no standby that sits idle until it’s needed.
What this means for you as a user:
- No downtime for updates and reboots. We update and restart one node at a time. Each node leaves the load balancer first and finishes its open requests, and the other two keep serving.
- A failing node doesn’t take CodeFloe down. The load balancer notices within seconds and sends traffic to the remaining nodes. Only the requests already running on that node are affected, and they can be retried.
Getting this to work takes much more than spinning up multiple VMs or running Forgejo/Arbinel on a k8s cluster. In fact, to get everything working smoothly, we had to make quite a few patches within Arbinel to avoid duplicated executions of Webhooks or Cron jobs and more.
| Part | What runs |
|---|---|
| Proxy | HAProxy on all three servers behind a floating IP address (keepalived), plus regional edges in Singapore and Virginia |
| Forge | three nodes, each with Arbinel, its own HAProxy (rate limits, bot rules), sshd, a code-search index, the MCP server and the egress meter |
| Repositories | a CephFS file system across three servers: 3 monitors, an active and a standby metadata server, 6 storage daemons, three copies of every file |
| Database | PostgreSQL on three nodes with Patroni and etcd, PgBouncer and HAProxy in front of each |
| Sessions, cache, queues | Valkey: a primary, two replicas, three Sentinels and three HAProxy pods (k8s) |
| Issue search | Typesense Cluster on three nodes (k8s) |
| Files | object storage for attachments, LFS, avatars and packages, plus a separate store for repository archives |
Why search had to change. Forgejo’s built-in search indexes are files on one server, which doesn’t work when three servers handle requests:
- Issue and pull request search runs on Typesense, a separate search service that every node writes to and reads from. With one index per node, each would miss what was created on the other two. Forgejo does not support Typesense as of today.
- Code search stays on
zoekt, but every node keeps its own copy of the index, about 80 GB. zoekt reads its index directly from disk and needs local storage to be fast. A push is indexed at once by the node that received it, and the other nodes notice the new commit in the database within a minute and index it too.
How it’s orchestrated. Forgejo assumes it runs on a single server, so we had to patch Arbinel and teach it to run within a cluster group:
- Leader election. The nodes elect a leader through a lease in the database, so scheduled jobs such as mirror syncs and cleanups, and the incoming mailbox, run on exactly one node. Another node takes over within seconds if the leader goes away.
- Drain mode. A node tells the load balancer it’s leaving and finishes its open requests before it stops (#37). Our deployment automation uses this to update the nodes one at a time, and it stops all three together only for a database schema change, after checking it ahead of time.
- Automatic failover for the DB and Caches. The database fails over through Patroni, Valkey through Sentinel, the entry address through keepalived, and Ceph heals its own storage. Kubernetes keeps Valkey and Typesense running.
The full wiring is in the diagram below and in our infrastructure docs.
