Top all-time · #17744
Common Crawl
Company blog · blog.commoncrawl.orgdataprogramming
Common Crawl is the company blog at blog.commoncrawl.org, writing about data and programming. Since 2014, 4 different Hacker News readers have linked it 8 times, most often “Evaluating graph computation systems”. It is still writing: the last post was on 2026-10-22, and you can follow it by feed.
We build and maintain an open repository of web crawl data that can be accessed and analyzed by anyone.
Most-linked posts?
- Evaluating graph computation systems2 people · 3 mentions · first linked 2015-04-01
What Hacker News says?
If anyone is interested in graph processing at scale, the largest Web Data Commons Hyperlink Graph[1] (created from Common Crawl[2] data) is 3.5 billion web pages and 128 billion hyperlinks. To my knowledge, it's the largest freely available real world based graph dataset. Numerous graph systems have been made that can scale to this data, frequently on just a single powerful node, including FlashGraph[3] (using SSDs…
Comment linking /2015/04/evaluating-graph-computation-systems-performance-and-scale/
on “One trillion edges: graph processing at Facebook scale [pdf]”Submitted Web image size prediction for efficient focused image crawling as a story · 16 points
HN discussion[lightly modified version of a comment I put on the article as I love HN for discussion!] Great article -- we're excited there's so much interest in the web as a dataset! I'm part of the team at Common Crawl and thought I'd clarify some points in the article. The most important is that you can download all the data that Common Crawl provides completely for free, without the need to pay S3 transfer fees or process it…
Comment linking /2014/11/october-2014-crawl-archive-available/
on “Wouldn't it be fun to build your own Google?”Submitted Evaluating graph computation systems as a story · 4 points
HN discussionAbsolutely spot on - between FlashGraph and Frank McSherry's COST work, the two have really pushed the envelope on efficient large scale graph analysis. Frank McSherry wrote a "call to arms" for the broader graph community at [1]. The main point of interest is that academia generally compared their work with existing distributed graph processing systems, celebrating when any achievements were made, yet not aware of…
Comment linking /2015/04/evaluating-graph-computation-systems-performance-and-scale/
on “Processing Billion-Node Graphs on an Array of Commodity SSDs”Submitted February 2016 Common Crawl Archive Now Available as a story · 1 points
HN discussionI cannot recommend FlashGraph strongly enough. FlashGraph was one of the first graph computation engines to enable near trivial analysis of the Web Data Commons Hyperlink Graph - 3.5 billion web pages and 128 billion links. For smaller graphs, analysis using FlashGraph is hilariously quick. If you're interested in how this is achieved, refer to [1]. From memory, Da Zheng said he created FlashGraph primarily as he wa…
Comment linking /2015/02/analyzing-a-web-graph-with-129-billion-edges-using-flashgraph/
on “Processing Billion-Node Graphs on an Array of Commodity SSDs”Submitted Analyzing a Web graph with 129B edges using FlashGraph as a story · 1 points
HN discussion