Top all-time · #22496
MERJ
Company blog · merj.com/blogwebdatabusiness
MERJ is the company blog at merj.com, writing about web, data and business. Since 2023, 3 different Hacker News readers have linked it 7 times, most often “Show HN: Removing 30B Resource Requests Increases HTML Fetches by 22%”. No feed was found, so read it on the site itself.
Too few sampled comments (2) to say how often HN recommends it.?
Why HN links it
HN readers cite this MERJ investigation of Reddit's robots.txt, which reports that Reddit serves different robots.txt files to Google and to other user agents or IP addresses. Commenters use it as evidence.
Summarised by Claude from a sample of 2 HN comments that link here. Each comment is marked recommending, critical or neutral (citing without judging); how this works.
We create engineering advantage for enterprises worldwide. Shaping platforms that are discoverable by good bots, search engines, and customers.
Most-linked posts?
- Show HN: Removing 30B Resource Requests Increases HTML Fetches by 22%2 people · 2 mentions · first linked 2023-11-30
What Hacker News says?
They serve a different robots.txt to Google: https://merj.com/blog/investigating-reddits-robots-txt-cloak... You can see it here: https://search.google.com/test/rich-results/result?id=_mYogl... (click on "View Tested Page")
Comment linking /blog/investigating-reddits-robots-txt-cloaking-strategy
on “Google is the only search engine that works on Reddit now, thanks to AI deal”Submitted Agent interactions are failing silently as a story · 6 points
HN discussionRelated thread for the official blog post: https://news.ycombinator.com/item?id=40799275 Side note: They seem to serve other robots.txt for different User-Agents & IPs: https://merj.com/blog/investigating-reddits-robots-txt-cloak...
Comment linking /blog/investigating-reddits-robots-txt-cloaking-strategy
on “Reddit's Robots.txt Changed”Submitted Show HN: Removing 30B Resource Requests Increases HTML Fetches by 22% as a story · 6 points
HN discussionSubmitted Accept-Language Redirects Could Be Blocking Search Engines and AI Crawlers as a story · 1 points
HN discussionSubmitted Discovering and Diagnosing a Google Adsense Rendering Bug as a story · 1 points
HN discussionSubmitted Removing 30B Resource Requests Increases HTML Fetches by 22% as a story · 1 points
HN discussion