@mattwelke First, it’s very kind to ask this, not many people care for that!
(First I wanna point out that the following opinions are my own and do not represent CF as a project)
So articles like the CB one (and others which bring these up) always suffer from a few problems:
- They are unspecific in what “load” really means here and what actions (at scale) really harm them or are considered harmful in general
- Them being written by people who don’t wanna use AI - or never have - and hence might objectively lack the insights how AI really operates under the hood
- Them being annoyed by additional infra work on their end to adapt in any kind of way to the LLM area, being it to block something entirely or to “adapt” to the new reality
@mcepl has already raised a few good and important points about the topic.
There are a few shades to the use of AI in general, i.e. whether it’s about active personal use of model for inference (i.e. individuals trying to generate code) or activities being used for model training.
For the former, if one interacts on a repo with AI as a developer, they usually check out the repo once and then use the local checkout for the active work. This is usually concluded by a PR in which the model (usually) puts more information than needed.
In this work, the interaction with the platform is rather low and AI might even use less requests than a human who would develop this feature and needs to browse the codebase or the issue tracker multiple times until the required information is found.
The same goes for replying to comments, doing reviews, etc. Overall, AI only does remote requests if it needs a fresh update of the current state or needs to check anything otherwise (e.g. new releases). In summary, this should be less requests in total than a human who would do the same work. Of course, a human would take much longer, so the requests would be spread across a longer time period.
The point(s) where AI is certainly causing more load are tests and CI. AI is writing much more tests and hence the code base grows a lot over time and there are more tests being run in CI.
The tests might again do interactive checks or queries or downloads within them, leading to additional requests. In former times, those tests simply did not exists as nobody wrote them (humans are lazy) or they were too complicated.
The areas that are affected here are storage and compute. Storing unnecessary details in PR descriptions and/or uploading the plan documents from AI bloats the repo. Once it, the commit exists forever and will be carried forward. Deleting it later will trim the repo a bit but the actions to create and delete still exist.
For CF as a platform, this means the following: CICD, resp. compute time, must be limited and billed, as otherwise it will just explode both in usage and cost.
Tests are important but users should then either run them on hardware of theirs (i.e. own runners) or value the time spent on hosted hardware and pay for the usage. That’s somewhat easy to control on the platform level with limits and the overall available resources.
Storage is the other point. The more commits, the more PRs, the larger the repository size and DB grows. Every action is eventually a DB transaction.
Users can be wary here by
- using squash merges instead of rebase merging commits (which carries along all indiv commits of a PR).
- Ensuring that no plan documents or other unneeded one-time assets are committed to the main branch (or even PR branch)
- Ensuring that dependency updates via renovate & friends don’t bump every individual dependency daily but only every few weeks (critical CVEs are usually raised anyhow out of schedule)
These bring down both the commit count and the amount and size of stored files. This really makes a difference for a platform, especially if many people adhere to these basic approaches.
Overall though, platforms like CF do exist to help with building! That means, they exist for that and try to optimize the handling of user interactions and the storage of a lot (and frequent) code contributions. CF wants to help building new tools and serve as the main interaction layer for devs.
If the underlying hardware and the configuration of the platform is “good”, there is a lot of headroom and users shouldn’t need to worry about their individual usage, at least not until they’ve reached the free limits.
And even beyond that, higher usage can be handled by compensating for it financially, which means nothing else that the platform can afford more resources (=hardware) for the required load.
At this point I haven’t commented on the first point of AI, i.e. the training. This is what actually matters a load more in terms of “load” if one does not restrict it. And this is what CF actually does. Without it, the site wouldn’t be snappy but drowning under multiple hundreds of requests per second from bots scanning repos for every individual commit.
CF does not do this to prevent AI but we also don’t allow “unlimited” or “ungraceful” use of it. Neither do we use any kind of existing data for internal training or similar. AFAIK GH does do both and hence also has a lot higher load therefore (aside from the “normal” Git operations load they have).
To give some insights for the load we currently have: CF as a platform has around 50 requests per second. HAProxy can easily take 100k+ per second.
The problem is not the proxy here but how much the underlying Git server can then take in terms of Git requests - and adjacent DB requests (not every request is directly a Git request). Also it differs whether such one can be answered from existing cache or needs a full query deep down in the cold archive).
So all in all, there’s a lot to say about this topic and it’s not so “easy” to state that AI is causing a lot of “bad” load on a platform no matter what. I would even go so far to say that normal websites or other non-optimized public web content is way more “harmed” than developer platforms (as these ideally have limits in place, use caching and filter for authenticated/unauthenticated traffic).