I am experiencing 403 Forbidden errors on Chrome-based browsers (on both my Chromebook and Android device). The issue appears to be tied partly to the User Agent, as spoofing the Firefox user agent on the Chromebook eliminates that issue but exposes another one: extremely sensitive rate limits (to the point of not being able to log in due to 429 errors). These errors affect the main site, this forum, and the status page.
I was able to log in using Firefox on my phone to post this and did not encounter any issues there. The curl command on the Chromebook’s Linux environment also seems to be unaffected. Using a VPN does not resolve the issue so it is likely not tied to my IP address. Online multi-browser testing tools seem to confirm the issue.
Please let me know if I can provide any additional information.
@binanary Thanks for reporting (and sorry for the issues!)
Yes, we made some User-Agent related proxy ACL changes two days ago after we had one scraper masking itself with a UA that wasn’t caught by the existing protection patterns.
However, it seems that the adjustments were too broad and we’re now also catching valid UA with that, especially chrome-based ones. We got another report earlier that day.
One get’s 429 after too many 4xx requests happened, which will put the IP into a ban for a few hours.
If possible, can you share the UAs of the affected browsers and/or send us your IP (in a private message), so we can see which UA were blocked by accident and ensure that these aren’t included anymore in the future?
Unfortunately the whole game of catching scrapers which try to mask themselves as “valid” users and catching them while still allowing “real” users isn’t an easy one. We have the believe that we can find a good middle way with more information over time and avoid tools like Anubis and others.
1 Like
Everything seems to be working now! I passed along the details you requested in a private message