Hi there, I moved to codefloe recently. Doing any git commands with my repos on codefloe is extremely slow right now. I did not have this problem before today. The easiest way to reproduce would be to try and clone any repo on codefloe.
git clone command showed that I am only getting a transfer rate of 40 kbps. Other commands like git push and git fetch also take extremely long (~10 seconds). git fetch also failed once with “ssh: Could not resolve hostname codefloe.com”
I do not have this issue with github currently. I also tried a different device and a vpn and the issue persists.
We’ve noticed this as well already, it is caused by a widespread crawling of facetted endpoints which are heavy on the Git backend side (?assignee=0&labels=910,911,915,938,940&milestone=-1&poster=0&project=0&sort=leastupdate&state=open&type=all")
In the last 10m alone, we had 1.3k requests from multiple hundred unique IPs to paths like the above one. We’re currently working on a dynamic ACL to block these requests in a meaningful way, while not limiting actual users.
On top, we’ve also identified a registered account doing authenticated crawling across the platform. We have contacted them to ask them to stop and for clarification.
On top, we’ve also added a new rate-limiting for authenticated widespread crawling and hope that this will prevent future cases of this. Please let us know in case you are affected by a false-positive detection of these new measures!
@codefloe Thank you for the update. Happy to hear that the team is already addressing it. I am still experiencing the same issue right now. Hope it can be fixed soon!
Metrics don’t shown any slowness during that time period. The measures we applied work as expected on the proxy side.
We currently are not working on any additional changes. If you still experience slow operations, it might be most likely by a factor on your end (or the connection in between yours and our servers).
If you want us to inspect in more detail, you need to provide the exact timestamps of your operations and other potential information to identify your requests.
More testing indicated that it was only slow while doing git operations over ssh.
For example git clone https://codefloe.com/bevane/test was fast while git clone ssh://git@codefloe.com/bevane/test.git was painfully slow.
Also doing ssh -T git@codefloe.com was unresponsive while ssh -T git@[insert ip address of codefloe.com here] immediately responded.
It is likely a DNS issue. I managed to get around it by forcing my ssh connections to ipv4 by adding AddressFamily inet to ~/.ssh/config
Leaving up the details in case anyone else experiences the same issue. @codefloe This thread can be marked as solved, thanks!
