Top all-time · #6493
Joey Lynch
Personal blog · jolynch.github.iosystemsprogramming
Joey Lynch's personal blog at jolynch.github.io covers systems and programming. Since 2020, 12 different Hacker News readers have linked it 20 times, most often “Use fast data algorithms (2021)”. The last post was in 2023, but the archive is still widely read.
Too few sampled comments (4) to say how often HN recommends it.?
Why HN links it
Linked for practical overviews of fast data algorithms, especially choices of hashing and compression. Commenters cite it as a reference for which algorithms to use instead of slower ones.
Summarised by Claude from a sample of 4 HN comments that link here. Each comment is marked recommending, critical or neutral (citing without judging); how this works.
Most-linked posts?
- Use fast data algorithms (2021)7 people · 12 mentions · first linked 2022-05-02
- Distributed Systems Shibboleths2 people · 3 mentions · first linked 2022-04-25
- SemVer Considered Harmful2 people · 2 mentions · first linked 2020-07-05
What Hacker News says?
Here is a nice page for mapping "what first came to mind" to "what's actually fastest in 2021": https://jolynch.github.io/posts/use_fast_data_algorithms/ TL;DR, lz4 is 10x faster than gzip (though does not produce as small as a file), zstd is twice as fast and better.
Comment linking /posts/use_fast_data_algorithms/
on “Cron-based backup for SQLite”Submitted Distributed Systems Shibboleths as a story · 360 points
HN discussionThis was on HN a while ago https://jolynch.github.io/posts/use_fast_data_algorithms/ and reaches the same conclusion. I am inclined to use Zstd even in cases where people would use lz4. Zstd is very fast.
Comment linking /posts/use_fast_data_algorithms/
on “Bzip3: A spiritual successor to BZip2”Submitted Use fast data algorithms (2021) as a story · 221 points
HN discussionChromium is kind of stuck with zlib because it's the algorithm that's in the standards, but if you're making your own protocol, you can do even better than this by picking a better algorithm. Zstandard is faster and compresses better. LZ4 is much faster, but not quite as small. Some reading: https://jolynch.github.io/posts/use_fast_data_algorithms/ (As an aside, at my last job container pushes / pulls were in the de…
Comment linking /posts/use_fast_data_algorithms/
on “Zlib-rs is faster than C”Submitted Supermarkets and Efficient Queueing (2019) as a story · 4 points
HN discussionThe article already mentions zstd but for an IMO simple and accurate overview take a look at the recommendations from https://jolynch.github.io/posts/use_fast_data_algorithms/ .
Comment linking /posts/use_fast_data_algorithms/
on “Zstandard Worked Example”Submitted Use Fast Data Algorithms (2021) as a story · 3 points
HN discussionEvery zone has a copy, and clients always read their local zone copy (via pooled memcached connections) first and fallback only once to another zone on miss. Key is staying in zone and memcached protocol plus super fast server latencies. It's been a little while since we measured, but memcached has a time to first byte of around 10us and then scales sublinearly with payload size [1]. Single zone latency is variable…
Comment linking /pdf/wlllb-apachecon-2022.pdf
on “Netflix's Distributed Counter Abstraction”Submitted SemVer Considered Harmful as a story · 3 points
HN discussionNo, most software engineers think salespeople are idiots because they have a bad habit of selling things to customers that don't exist yet, and in many cases cannot exist, and then you have to tell them, point blank, that they just lied to a customer to secure a sale and no solution to their problem with be coming. Salespeople generally only care about this quarter, if a chargeback or contract cancelation happens, t…
Comment linking /posts/distsys_shibboleths/
on “The Negative Link Between Extraversion and Perceived Listening”Submitted Use Fast Data Algorithms (2021) as a story · 2 points
HN discussionsha256 is a very slow algorithm, even with hardware acceleration. BLAKE3 would probably make a noticeable performance difference. Some reading from 2021: https://jolynch.github.io/posts/use_fast_data_algorithms/ It is really hard to describe how slow sha256 is. Go sha256 some big files. Do you think it's disk IO that's making it take so long? It's not, you have a super fast SSD. It's sha256 that's slow.
Comment linking /posts/use_fast_data_algorithms/
on “I made my own Git”Submitted Distributed Systems Shibboleths as a story · 2 points
HN discussionDon't forget that md5 is comparatively slow & there are way options for hashing nowadays: https://jolynch.github.io/posts/use_fast_data_algorithms/
Comment linking /posts/use_fast_data_algorithms/
on “Bear Does Analytics with CSS”Submitted SemVer Considered Harmful as a story · 2 points
HN discussionA few weeks ago, I decided to lock myself in my apartment and write a build system. I have always felt like developers wait ages for Docker, and I wanted to see why. (Cycle times on the app I develop are a minute for my crazy combination of shell scripts that I use, or up to 5 minutes for what most of my teammates do. This is incomprehensibly unproductive.) It turns out, it's all super crazy at every level. Things l…
Comment linking /posts/use_fast_data_algorithms/
on “Intel Xeon Max 9480 Deep-Dive 64GB HBM2e Onboard Like a GPU or AI Accelerator”Submitted Use fast data algorithms (2021) as a story · 1 points
HN discussionThere are a variety of use cases that dictate which algorithm is going to perform best. For example, you might use Zstandard -19 if you are compressing something once and transferring it over a slow network to millions of people. You might use LZ4 if you are generating a unique large piece of data interactively for thousands of concurrent users, because it compresses faster than Zstandard. Basically, if you're const…
Comment linking /posts/use_fast_data_algorithms/
on “Data Compression Drives the Internet. Here’s How It Works”Submitted Wait Before You Sync as a story · 1 points
HN discussion