Top all-time · #13598
Read the Docs
Company blog · about.readthedocs.com/blogprogrammingweb
Read the Docs is the company blog at about.readthedocs.com, writing about programming and web. Since 2024, 5 different Hacker News readers have linked it 12 times, most often “AI crawlers need to be more respectful”. It is still writing: the last post was on 2026-09-08, and you can follow it by feed.
Too few sampled comments (4) to say how often HN recommends it.?
Why HN links it
HN readers link this blog for its posts on bot and crawler traffic overwhelming websites, including Read the Docs' own bandwidth bills and a DDoS attack on the site. Comments cite these as sources.
Summarised by Claude from a sample of 4 HN comments that link here. Each comment is marked recommending, critical or neutral (citing without judging); how this works.
Read the Docs is a documentation building and hosting platform aimed at helping developers creating documentation from code with versioned documentation, integrated search, pull request previews and more.
Most-linked posts?
- AI crawlers need to be more respectful4 people · 7 mentions · first linked 2024-07-25
- /pricing/2 people · 2 mentions · first linked 2024-03-16
What Hacker News says?
Yep -- our story here: https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse... (quoted in the OP) -- everyone I know has a similar story who is running large internet infrastructure -- this post does a great job of rounding a bunch of them up in 1 place. I called it when I wrote it, they are just burning their goodwill to the ground. I will note that one of the main startups in the space worked with us direc…
Comment linking /blog/2024/07/ai-crawlers-abuse/
on “FOSS infrastructure is under attack by AI companies”Submitted AI crawlers need to be more respectful as a story · 226 points
HN discussionAnything they can, judging by the fact that they're hitting random endpoints instead of using those offered to developers. Similar thing happened to readthedocs[1] causing a surge of costs that nobody wants to answer for. In the readthedocs situation there was one case that was a bugged crawler causing it to try and scrape the same HTML files repeatedly to the tune of 75TB, could also be happening here with OSM (par…
Comment linking /blog/2024/07/ai-crawlers-abuse/
on “Dear AI Companies, instead of scraping OpenStreetMap, how about a $10k donation?”Submitted Understanding the recent DDoS attack against Read the Docs as a story · 217 points
HN discussionMy employer, Read the Docs, has a blog on the subject ( https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse... ) of how we got pounded by these bots to the tune of thousands of dollars. To be fair though, the AI company that hit us the hardest did end up compensating us for our bandwidth bill. We've done a few things since then: - We already had very generous rate limiting rules by IP (~4 hits/second sustai…
Comment linking /blog/2024/07/ai-crawlers-abuse/
on “AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders”Submitted 10 years of sustainable open source as a story · 4 points
HN discussionAuthor here. This was not a misconfigured data scraper. We see those every week[1]. This attack wasn't scraping useful content. It was almost entirely 404s and 302s and pulled virtually zero real docs. It specifically looked for URLs not served by the CDN and when it found a pattern, did millions of variations of it. Whether built by an AI or not, it was designed to cause outages and financial damage from autoscalin…
Comment linking /blog/2024/07/ai-crawlers-abuse/
on “Understanding the recent DDoS attack against Read the Docs”Submitted Read the Docs: 10 years of sustainable open source as a story · 2 points
HN discussionWe did get $7k out of one of the AI companies based on the massive bandwidth costs they caused us. https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse...
Comment linking /blog/2024/07/ai-crawlers-abuse/
on “OpenAI's bot crushed this seven-person company's web site 'like a DDoS attack'”This keeps happening -- we wrote about multiple AI bots that were hammering us over at Read the Docs for >10TB of traffic: https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse... They really are trying to burn all their goodwill to the ground with this stuff.
Comment linking /blog/2024/07/ai-crawlers-abuse/
on “OpenAI's bot crushed this seven-person company's web site 'like a DDoS attack'”If you're looking to host documentation, Read the Docs has been hosting docs for open source projects for 15 years now (I'm one of the co-founders). We put a small, ethical ad on your docs pages as tradeoff, and to make it sustainable, along with hosting credits from AWS (Thanks AWS!). We're happy to help host docs if that's what you're looking for: https://about.readthedocs.com/pricing/#/community
Comment linking /pricing/
on “Vercel ends open-source sponsorship program giving projects 24hr notice”I agree, this doesn't look great, but there is still a free tier: https://about.readthedocs.com/pricing/#/community
Comment linking /pricing/
on “Ask HN: ReadTheDocs Became Proprietary Now?”