Top all-time · #15586
Lumar
Company blog · deepcrawl.com/blogwebbusiness
Lumar is the company blog at deepcrawl.com, writing about web and business. Since 2016, 5 different Hacker News readers have linked it 5 times. It is still writing: the last post was on 2026-10-19, and you can follow it by feed.
Too few sampled comments (3) to say how often HN recommends it.?
Why HN links it
HN readers link Lumar's blog as a reference on robots.txt directives, such as whether Noindex and Disallow work together. Other links point to its terms of use, not to its writing.
Summarised by Claude from a sample of 3 HN comments that link here. Each comment is marked recommending, critical or neutral (citing without judging); how this works.
The ultimate website optimization platform with tools for tech SEO, GEO, web accessibility, site speed, & more, built for enterprise scale.
What Hacker News says?
I've been in disagreements with SEO people quite frequently about a "Noindex" directive for robots.txt. There seem to be a bunch of articles that are sent to me every time I question its existence[0][1]. Google's own documentation says that noindex should be in the meta HTML but the SEO people seem to trust these shady sites more. I haven't read through all of the code but it assuming this is actually what's running…
Comment linking /blog/best-practice/robots-txt-noindex-the-best-kept-secret-in-seo/
on “Google’s robots.txt parser is now open source”Submitted Search engines and social media crawlers render JavaScript as a story · 4 points
HN discussionI do not have a disallow set for this page but based off of this page it seems perhaps I might have to set one: https://www.deepcrawl.com/blog/best-practice/noindex-disallo... Specifically the part: >Noindex (robots.txt) + Disallow: This prevents pages appearing in the index, and also prevents the pages being crawled. However, remember that no PageRank can pass through this page.
Comment linking /blog/best-practice/noindex-disallow-nofollow/
on “Ask HN: How to stop Google indexing dynamic search pages?”Submitted DeepCrawl: Delivering SEO Confidence as a story · 2 points
HN discussionWe may in fact be charging too little...as we have not had enough complaints about it ;-) Our prices will be increasing on most plans +25% in a few days, now that we just launched a powerful suite of features. We looked at other crawling apps in our space, and found that they too didn't offer refunds: https://www.deepcrawl.com/terms-of-use/ https://www.semrush.com/company/legal/refund-policy/ Although this one is a…
Comment linking /terms-of-use/
on “Ask HN: How do you handle Refund Requests for your SaaS?”