Top all-time · #4498
quarto-inputc42a213491699d28
Personal blog · livefreeordichotomize.comdataprogramming
quarto-inputc42a213491699d28 is a personal blog at livefreeordichotomize.com about data and programming. Since 2017, 17 different Hacker News readers have linked it 17 times, most often “Using AWK and R to parse 25TB”. It is still writing: the last post was on 2026-08-03, and you can follow it by feed.
Too few sampled comments (4) to say how often HN recommends it.?
Why HN links it
HN readers cite this blog's 2019 post on using awk to process multi-terabyte datasets as evidence that Unix text tools can handle big data without distributed systems. The comments link it as an example rather than praise it.
Summarised by Claude from a sample of 4 HN comments that link here. Each comment is marked recommending, critical or neutral (citing without judging); how this works.
Most-linked posts?
- Using AWK and R to parse 25TB6 people · 6 mentions · first linked 2019-06-06
- Using Awk and R to parse 25tb (2019)3 people · 3 mentions · first linked 2022-10-16
- Using Awk and R to parse 25tb (2019)2 people · 2 mentions · first linked 2023-09-08
What Hacker News says?
> How would running awk for analysis on 6TB of data work quickly and efficiently? In that 6TB is not that huge of an amount That's their total dataset, and there's no "real time" requirement. They can start a batch process, process the data, and be done with it. Here's an example of someone using awk (read further down for the relevant section): https://livefreeordichotomize.com/posts/2019-06-04-using-awk... "I was…
Comment linking /posts/2019-06-04-using-awk-and-r-to-parse-25tb/
on “Big data is dead (2023)”Submitted Using AWK and R to parse 25TB as a story · 312 points
HN discussionbefore you go down the spark route, consider perl/unix-tools may do this kind of thing quite well: https://livefreeordichotomize.com/2019/06/04/using_awk_and_r...
Comment linking /2019/06/04/using_awk_and_r_to_parse_25tb/
on “OpenRefine – free, open source, powerful tool for working with messy data”Submitted Hill for the data scientist: an xkcd story as a story · 122 points
HN discussionI love Unix pipes and I love the immense computational power you can achieve through streaming pipelines. You hear stories of people processing terabytes of data faster and cheaper than a distributed cloud solution, by the virtue of Unix pies[1]. But I don’t really see them in any other environment. I can’t think of anything that uses something similar except for big, distributed data processing pipelines like Apach…
Comment linking /posts/2019-06-04-using-awk-and-r-to-parse-25tb/
on “Show HN: The Unix Pipe Card Game – teach kids basic Unix commands”Submitted Using Awk and R to parse 25tb (2019) as a story · 12 points
HN discussionAWK/Sed are actually very efficient. See this post for example, it describes parsing 25Tb of data with awk: https://livefreeordichotomize.com/posts/2019-06-04-using-awk...
Comment linking /posts/2019-06-04-using-awk-and-r-to-parse-25tb/index.html
on “My Node.js is a bit Rusty”Submitted Using Awk and R to parse 25 TB as a story · 5 points
HN discussionSubmitted Using Awk and R to parse 25tb as a story · 4 points
HN discussionSubmitted Using Awk and R to parse 25tb of DNA sequencing data as a story · 3 points
HN discussionSubmitted Using Awk and R to parse 25tb (2019) as a story · 2 points
HN discussionSubmitted Show HN: Building a data-driven CV with R as a story · 2 points
HN discussionSubmitted Live Free or Dichotomize – Using Awk and R to Parse 25tb as a story · 2 points
HN discussionSubmitted Understanding Propensity Score Weighting as a story · 1 points
HN discussionSubmitted Using Awk and R to parse 25tb as a story · 1 points
HN discussionSubmitted Understanding propensity score weighting as a story · 1 points
HN discussionSubmitted Why you maybe shouldn't care about that p-value as a story · 1 points
HN discussion
