On 24/11/2025 15:04, Windl, Ulrich wrote:
Hi!
I think I would have tried to return a "please crawl elsewhere!" message to
these crawlers instead.
... or do what (sadly) many other sites are doing these days - implement
https://github.com/TecharoHQ/anubis.
Honza
What if the crawlers use an account now?
Kind regards,
Ulrich Windl
-----Original Message-----
From: Users <[email protected]> On Behalf Of Reid Wahl
Sent: Tuesday, November 18, 2025 1:51 AM
To: Cluster Labs - All topics related to open-source clustering welcomed
<[email protected]>
Subject: [EXT] [ClusterLabs] Login is required for now on
projects.clusterlabs.org
A while back, we made tasks, commits, and source files on
projects.clusterlabs.org publicly accessible by default.
Lately we've been inundated with HTTP requests from AI crawler bots
(from many IP addresses) that ignore robots.txt. To mitigate the
effects, we've set the permissions to "All Users" for the apps that
access tasks, commits, and source files. This means that no matter
what the permissions are for any particular object, you are now
required to log in before you can view it.
It's easy to create an account, and you don't need any additional
permissions to view most things once you're logged in.
We will probably continue requiring a login to view commits and source
code. I can't think of any reason those need to be publicly visible on
projects.clusterlabs.org, since they're already visible on GitHub.
We hope to make the tasks/bug reports visible without a login again in
the future.
--
Regards,
Reid Wahl (He/Him)
Senior Software Engineer, Red Hat
RHEL High Availability - Pacemaker
_______________________________________________
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users
ClusterLabs home: https://www.clusterlabs.org/
_______________________________________________
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users
ClusterLabs home: https://www.clusterlabs.org/
_______________________________________________
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users
ClusterLabs home: https://www.clusterlabs.org/
