Top all-time · #2425
Zyte #1 Web Scraping Service
Company blog · blog.scrapinghub.comdataprogrammingbusiness
Zyte #1 Web Scraping Service is the company blog at blog.scrapinghub.com, writing about data, programming and business. Since 2012, 31 different Hacker News readers have linked it 128 times, most often “Using Git to manage staff vacations in a large distributed team”. It is still writing: the last post was on 2026-11-02, and you can follow it by feed.
Too few sampled comments (10) to say how often HN recommends it.?
Why HN links it
HN readers link Scrapinghub's blog for web scraping material: Scrapy, Splash, Portia, ScrapyRT, clean article extraction, and company history. Commenters note it is tied to the company's own products.
Summarised by Claude from a sample of 10 HN comments that link here. Each comment is marked recommending, critical or neutral (citing without judging); how this works.
Insights, tutorials and analysis on web scraping and data extraction — written by the people closest to the data.
Most-linked posts?
- Using Git to manage staff vacations in a large distributed team5 people · 5 mentions · first linked 2015-06-08
- Vizlegal: Rise of Machine-Readable Laws and Court Judgments3 people · 3 mentions · first linked 2016-01-13
- Migrate Your Kimono Projects to Portia2 people · 3 mentions · first linked 2016-02-28
- Christmas Eve vs. New Year’s Eve: Last Minute Price Inflation?2 people · 2 mentions · first linked 2016-01-06
- Cyber Monday: How to Build Your Own Price Monitoring Tool2 people · 2 mentions · first linked 2016-11-24
- Distributed Frontera: Web Crawling at Scale2 people · 2 mentions · first linked 2015-08-13
- Introducing Scrapy Cloud with Python 3 Support2 people · 2 mentions · first linked 2016-08-17
- Show HN: Portia2Code — Turn Portia Spiders into Scrapy Spiders2 people · 2 mentions · first linked 2016-06-29
- How Web Scraping Is Revealing Lobbying and Corruption in Peru2 people · 2 mentions · first linked 2016-03-09
- Scrapy Tips from the Pros: how to debug your spiders2 people · 2 mentions · first linked 2016-05-18
What Hacker News says?
Haven't tried this[0] yet, but Scrapy should be able to handle JavaScript sites with the JavaScript rendering service Splash[1]. scrapy-splash[2] is the plugin to integrate Scrapy and Splash. [0] https://blog.scrapinghub.com/2015/03/02/handling-javascript-... [1] https://splash.readthedocs.io/en/stable/index.html [2] https://github.com/scrapy-plugins/scrapy-splash
Comment linking /2015/03/02/handling-javascript-in-scrapy-with-splash/
on “Ask HN: What are best tools for web scraping?”Submitted How Web Scraping Is Revealing Lobbying and Corruption in Peru as a story · 398 points
HN discussionNot sure if you're interested, but we (Scrapinghub) do offer a Kimono to Portia migration https://blog.scrapinghub.com/2016/02/25/migrate-your-kimono-... Otherwise, I'd recommend you check out Portia (open source). We're in the middle of releasing the beta 2.0 version.
Comment linking /2016/02/25/migrate-your-kimono-projects-to-portia/
on “How Web Scraping Is Revealing Lobbying and Corruption in Peru”Submitted Portia, an open-source visual web scraper as a story · 367 points
HN discussionAll sorts of careers, for example: - developers who want to develop some data-based product (a travel agency website, who finds the best deals from airline companies); - lawyers can use it to structure the data from Judgments and Laws (so that they are able to query the data for things like: which judges have interpreted this law in their judgments) (more on this: http://blog.scrapinghub.com/2016/01/13/vizlegal-rise…
Comment linking /2016/01/13/vizlegal-rise-of-machine-readable-laws-and-court-judgments/
on “Scrapy Tips from the Pros”Submitted Scrapy Tips from the Pros as a story · 269 points
HN discussionThere is ScrapyRT: http://blog.scrapinghub.com/2015/01/22/introducing-scrapyrt-... In the project I work on we do have the usual periodic crawls and use ScrapyRT to let the frontend trigger realtime scrapes of specific items, all of this using the same spider code. Edit: Worth nothing that we trigger the realtime scrapes via AMQP.
Comment linking /2015/01/22/introducing-scrapyrt-an-api-for-scrapy-spiders/
on “Scrapy Tips from the Pros”Submitted Lessons learned scraping 100B product pages as a story · 265 points
HN discussionDisclaimer, I wrote the article. > I'm not fond of the implication at the end that scraping is justifiable because old websites are dinosaurs without APIs, and those websites are jerks for not doing so, and therefore scraping is the moral thing to do. It was not my intention to give that implication. The main implication behind CityBikes is that public services should already provide this information since, well, it…
Comment linking /2016/03/30/web-scraping-to-create-open-data/
on “Web Scraping to Create Open Data”Submitted Python 3 comes to Scrapy as a story · 249 points
HN discussionWe've actually developed a way that you can convert Portia projects into Scrapy spiders: https://blog.scrapinghub.com/2016/06/29/introducing-portia2c... and since this is all open source, here's a link to GitHub: https://github.com/scrapinghub/portia2code
Comment linking /2016/06/29/introducing-portia2code-portia-projects-into-scrapy-spiders/
on “Scrapely: The brains behind Portia, our visual web scraping tool”Submitted Web Scraping to Create Open Data as a story · 176 points
HN discussionScrapinghub writes some useful blog posts at https://blog.scrapinghub.com/ . It obviously has to do with using their frameworks and services, so it may not be very useful in your case.
Comment linking /
on “Ask HN: Advanced web crawling resources?”Submitted How to Crawl the Web Politely with Scrapy as a story · 139 points
HN discussionThere are AI solutions that can work on the HTML and manage to avoid the ads. Checkout https://blog.scrapinghub.com/extracting-clean-article-html-w...
Comment linking /extracting-clean-article-html-with-autoextract
on “Show HN: Eat the News – A text extraction playground for global RSS feeds”Submitted Spidermon: Scrapinghub’s Now Open Sourced Spider Monitoring Library as a story · 115 points
HN discussionHow a colleague of mine managed to get internet access in rural Uruguay https://blog.scrapinghub.com/2016/04/27/a-not-so-short-story...
Comment linking /2016/04/27/a-not-so-short-story-on-getting-decent-internet-access/
on “Ask HN: How do i build my own network to access internet in a rural area?”Submitted Web Scraping Finds Price Fluctuation as a story · 104 points
HN discussionScrapingHub was something I was interested in - as an example of opensource project which the founders used as a base to build a company. Does it help to use their model for you - https://blog.scrapinghub.com/2015/03/16/history-of-scrapingh...
Comment linking /2015/03/16/history-of-scrapinghub
on “Ask HN: Should I open-source my bootstrapped product?”Submitted Scrapely: The brains behind Portia, our visual web scraping tool as a story · 102 points
HN discussionSubmitted Why MongoDB is a bad choice for storing our scraped data as a story · 86 points
HN discussionSubmitted Portia: Open-Source Alternative to Kimono Labs as a story · 53 points
HN discussionSubmitted Scrapy Tips from the Pros: February 2016 Edition as a story · 49 points
HN discussionSubmitted XPath tips from the web scraping trenches as a story · 40 points
HN discussionSubmitted Show HN: Scrapy Real Time as a story · 37 points
HN discussionSubmitted Building Blocks of an Unstoppable Web Scraping Infrastructure as a story · 36 points
HN discussionSubmitted Show HN: Portia2Code — Turn Portia Spiders into Scrapy Spiders as a story · 31 points
HN discussionSubmitted Aduana: Link Analysis to Crawl the Web at Scale as a story · 24 points
HN discussionSubmitted Handling JavaScript in Scrapy with Splash as a story · 23 points
HN discussionSubmitted Optimizing memory usage of scikit-learn models using succinct Tries as a story · 23 points
HN discussion
