1362 points•705 comments•3d ago•by zdw•people.kernel.org ↗

Creepy Crawlies

Git.kernel.org handles about 6 million daily requests, with roughly two percent from legitimate developers and the rest from automated crawlers; about 14–16 of the 90 cores across five geo-distributed nodes are dedicated to rendering commits for scrapers, about 20% of capacity. An Anubis challenge blocks many bots, but around a third still gets through, and operators are turning off features to limit crawlable URLs and gate expensive actions, which may reduce anonymous access.

Discussion (705)

No comments yet

Top
Trending
New
For You
Bookmarks
Dossier