Top all-time · #1518
RealTime Data Compression
Personal blog · fastcompression.blogspot.comprogrammingsystems
RealTime Data Compression is a personal blog at fastcompression.blogspot.com about programming and systems. Since 2012, 47 different Hacker News readers have linked it 63 times, most often “RealTime Data Compression: LZ4 Explained”. The last post was in 2019, but the archive is still widely read.
Sometimes recommended · 5 of 31 sampled comments recommend it?
Why HN links it
HN readers link Yann Collet's blog as the primary source on his compression work: FSE entropy coding, LZ4, zstd, benchmarks, and zstd's origins. Commenters cite it for explanations of those topics.
- Recommended for · 5 comments
- Called the most helpful source for a basic idea of FSE; a good old blog; the only source one commenter found on hardware counter activity.
Summarised by Claude from a sample of 31 HN comments that link here. Each comment is marked recommending, critical or neutral (citing without judging); how this works.
Development blog on compression algorithms
Most-linked posts?
- /2013/12/finite-state-entropy-new-breed-of.html7 people · 7 mentions · first linked 2014-12-29
- RealTime Data Compression: LZ4 Explained4 people · 4 mentions · first linked 2012-01-14
- Huffman revisited, Part 3 – Depth-limited tree3 people · 4 mentions · first linked 2015-07-30
- /3 people · 3 mentions · first linked 2016-04-25
- /p/compression-benchmark.html2 people · 3 mentions · first linked 2014-07-26
- Small Data Compression2 people · 2 mentions · first linked 2016-02-05
- Portability woes: Endianness and Alignment, Part 22 people · 2 mentions · first linked 2014-11-25
- Counting bytes fast2 people · 2 mentions · first linked 2014-09-27
- Zstandard Reaches Final Compression Format2 people · 2 mentions · first linked 2016-06-17
- Debunking the LZ4 "20 years old bug" myth2 people · 2 mentions · first linked 2014-06-27
What Hacker News says?
I think zstd is originally not a Facebook project. https://fastcompression.blogspot.com/2015/01/zstd-stronger-c... I believe Yann Collet, author of lz4, eventually went on to work for Facebook and finished it there.
Comment linking /2015/01/zstd-stronger-compression-algorithm.html?m=1
on “Sapling: A new source control system with Git-compatible client”Submitted XXH3 – a new speed-optimized hash algorithm as a story · 275 points
HN discussionYeah read about Kraken a while ago and found the encode.ru thread you pointed. Its unfortunate that we can't compare that side by side. Those guys seem another of those compression geniuses. Overall the world of compression is moving very fast and well these days. But even with that Yann might be the first. Yan's work on FSE is old [0] and if I'm reading this correctly his work on FSE is a mix of his own work and Ja…
Comment linking /2013/12/finite-state-entropy-new-breed-of.html
on “Smaller and faster data compression with Zstandard”Submitted Debunking the LZ4 "20 years old bug" myth as a story · 178 points
HN discussionI found his (Yann's) blog posts the most helpful for getting a basic idea of what was going on with FSE: https://fastcompression.blogspot.com/2013/12/finite-state-en... and the followup https://fastcompression.blogspot.fr/2014/01/fse-decoding-how...
Comment linking /2013/12/finite-state-entropy-new-breed-of.html
on “Smaller and faster data compression with Zstandard”Submitted Counting bytes fast as a story · 128 points
HN discussionWWDC video talking about it: https://developer.apple.com/videos/play/wwdc2015/712/ Apple's docs: https://developer.apple.com/library/mac/documentation/Perfor... Generic info on finite state entropy compression: http://fastcompression.blogspot.com/2013/12/finite-state-ent... The last link was submitted to HN: https://news.ycombinator.com/item?id=7040951 The repo's README seems pretty clear. Google provides the rest.…
Comment linking /2013/12/finite-state-entropy-new-breed-of.html
on “How are zlib, gzip and Zip related? (2013)”Submitted Accessing unaligned memory as a story · 70 points
HN discussionHere are a bunch of blogs from my RSS reader that I think are worth reading. Some of them haven't updated in a while, but the archives may be worth a look anyway. * Blogs with a math focus: http://blog.sigfpe.com http://bit-player.org http://mathr.co.uk/blog/ https://www.johndcook.com/blog/ * Blogs with a programming languages focus: https://existentialtype.wordpress.com https://bluishcoder.co.nz http://blog.ezyang.…
Comment linking /
on “Ask HN: What programming blogs do you follow?”- ot · 2018-02-17
Submitted When to use Dictionary Compression as a story · 64 points
HN discussion This lookahead is very similar to the "lazy matching" used in Lempel-Ziv compressors! https://fastcompression.blogspot.com/2010/12/parsing-level-1... Did you compare it to Buzhash? I assume gearhash is faster given the simpler per iteration structure. (also, rand/v2's seeded generators might be better for gear init than mt19937)
Comment linking /2010/12/parsing-level-1-lazy-matching.html
on “CDC File Transfer”Submitted Compiler Warnings as a story · 34 points
HN discussionxxHash is not really "new", it's been available since 2014. At least, that's the oldest Python bindings available on PyPI: https://pypi.org/project/xxhash/0.0.1/ Sure, as it developed, xxHash variants appear. The latest in the family is XXH3 and it's already being used at least since 2019: http://fastcompression.blogspot.com/2019/03/presenting-xxh3....
Comment linking /2019/03/presenting-xxh3.html?m=1
on “Meow Hash (2018)”Submitted When to use dictionary compression (2018) as a story · 31 points
HN discussionI tried a fairly obscure and technical query, "what's a Finite State Entropy Encoder?" and it came up with https://news.ycombinator.com/item?id=7040951 which is about "Finite State Entropy - A new breed of entropy coder" http://fastcompression.blogspot.com/2013/12/finite-state-ent... Neither google nor DDG found that. Pretty neat. :)
Comment linking /2013/12/finite-state-entropy-new-breed-of.html
on “Launch HN: Andi (YC W22) – Q&A based, ad-free, anti-spam search engine”Submitted Portability woes: Endianness and Alignment, Part 2 as a story · 27 points
HN discussionFor the second situation, I prefer the compress/transmit/decompress metric (as a function of transmit pipe speed), as described here: http://fastcompression.blogspot.com/p/compression-benchmark....
Comment linking /p/compression-benchmark.html
on “Apple Open-Sources its Compression Algorithm LZFSE”Submitted RealTime Data Compression: Working with Streaming as a story · 19 points
HN discussionLZ4 explained for those who got interested http://fastcompression.blogspot.com/2011/05/lz4-explained.ht...
Comment linking /2011/05/lz4-explained.html
on “The LZ4 introduced in PostgreSQL 14 provides faster compression”Submitted Strong types in C as a story · 15 points
HN discussionThis is explanation from author http://fastcompression.blogspot.com/2011/05/lz4-explained.ht...
Comment linking /2011/05/lz4-explained.html
on “The LZ4 introduced in PostgreSQL 14 provides faster compression”- ot · 2015-08-25
Submitted Fuzz Testing Zstandard as a story · 15 points
HN discussion Here are some software engineering blogs from my RSS reader in addition to the ones listed in TFA. I honestly don't know how over or underrated they are, but I guess that varies depending on which communities you're closer to. Mikola Lysenko https://0fps.net Graphics programming, geometry and algorithms. --- Steven Wittens http://acko.net Not underrated in any sense of the word, but I have to include it in case ther…
Comment linking /
on “Programming blogs”Submitted Huffman revisited as a story · 11 points
HN discussionDiscussed a couple years ago here: https://news.ycombinator.com/item?id=5037089 And the followup discussed here: https://news.ycombinator.com/item?id=5075370 I repost because I'm becoming more and more frustrated with C, and wondered if others feel the same. Not because it's too low-level, rather because it seems to be trying (with limited success) to move to being a higher level language. There are times when I rea…
Comment linking /2014/11/portability-woes-endianess-and.html
on “The Unreasonable Effectiveness of C (2013)”Submitted ZSTD – A stronger compression algorithm as a story · 10 points
HN discussionOn a purely compression algorithm front, the recently format-stabilized [1] zstd [2] beats the pants down Brotli, in terms of compression speed, decompression speed, and compression ratio. Zstd isn't some random effort either, but a more flexible (in terms of compression time-for-ratio) effort by the creator of the popular and insanely fast lz4 compressor[3]. [1] http://fastcompression.blogspot.com/2016/06/zstandard…
Comment linking /2016/06/zstandard-reaches-final-format.html
on “Lossless compression with Brotli”Submitted Huffman revisited, Part 3 – Depth-limited tree as a story · 5 points
HN discussionThere is mention in the blog of the creator about window size. It can be configured down to 4KB to it can operate in low memory environments: http://fastcompression.blogspot.com/ see under "Custom window sizes" section
Comment linking /
on “Better Compression with Zstandard”Submitted RealTime Data Compression: LZ4 Explained as a story · 4 points
HN discussion
