Top all-time · #395
Matt Mahoney's Home Page
Personal blog · mattmahoney.netprogrammingdata
Matt Mahoney's Home Page is a personal blog at mattmahoney.net about programming and data. Since 2009, 138 different Hacker News readers have linked it 202 times, most often “Large Text Compression Benchmark”. No feed was found, so read it on the site itself.
Rarely recommended · 3 of 40 sampled comments recommend it?
Why HN links it
HN readers link his pages as the reference for compression benchmarks (the Large Text Compression Benchmark), the free Data Compression Explained book, and the zpaq compressor.
- Recommended for · 3 comments
- Called the Data Compression Explained book 'excellent' and 'really good'; one commenter just says 'Just read this'
Summarised by Claude from a sample of 40 HN comments that link here. Each comment is marked recommending, critical or neutral (citing without judging); how this works.
Most-linked posts?
- Large Text Compression Benchmark25 people · 27 mentions · first linked 2012-01-30
- Large Text Compression Benchmark19 people · 21 mentions · first linked 2023-04-05
- Data Compression Explained by Matt Mahoney16 people · 19 mentions · first linked 2010-03-09
- Large Text Compression Benchmark15 people · 16 mentions · first linked 2019-04-17
- ZPAQ: Incremental Journaling Backup Utility and Archiver12 people · 12 mentions · first linked 2014-04-10
- Rationale for a Large Text Compression Benchmark (2009)10 people · 14 mentions · first linked 2013-06-14
- Large Text Compression Benchmark (2023)7 people · 8 mentions · first linked 2023-12-03
- Rationale for a Large Text Compression Benchmark (2009)4 people · 6 mentions · first linked 2023-04-15
- /barkley/3 people · 3 mentions · first linked 2013-03-29
- Data Compression Explained (2011)3 people · 3 mentions · first linked 2024-06-10
What Hacker News says?
I believe (from quick code inspection) that the symmetry in encode/decode performance for QOI is because it has to generate the hash-table on decode too. Normal fast encoders use some kind of hash table or search to find matches, but encode the offset of the match in the source rather than in the table. QOI is encoding the offset into the table, which gives much shorter offsets but means the decoder has to maintain…
Comment linking /dc/text.html
on “QOI: Lossless Image Compression in O(n) Time”Submitted Large Text Compression Benchmark as a story · 97 points
HN discussionThere is an award-winning compressor that uses many statistical models (and a 3-layered dense neural network) to compress the data losslessly: http://www.byronknoll.com/cmix.html Also there is a free book that contains descriptions of various common and exotic compression formats http://www.mattmahoney.net/dc/dce.html Overall I think we are yet to see the full potential of deep learning unleashed on data compression…
Comment linking /dc/dce.html
on “Lepton image compression: saving 22% losslessly from images at 15MB/s”Submitted Data Compression Explained as a story · 73 points
HN discussionA similar approach has also been developed for use with zpaq, a compression format which stores the decompression algorithm as bytecode in the archive: [The configuration] "jpg_test2" by Jan Ondrus compresses JPEG images (which are already compressed) by an additional 15%. It uses a preprocessor that expands Huffman codes to whole bytes, followed by context modeling. http://mattmahoney.net/dc/zpaqutil.html
Comment linking /dc/zpaqutil.html
on “Lepton image compression: saving 22% losslessly from images at 15MB/s”Submitted Large Text Compression Benchmark as a story · 69 points
HN discussionMight add zpaq (C++) - http://mattmahoney.net/dc/zpaq.html to the list.
Comment linking /dc/zpaq.html
on “Restic – Backups Done Right”Submitted Data Compression Explained (2011) as a story · 26 points
HN discussionFor text compression benchmarks it's done http://mattmahoney.net/dc/text.html Matt doesn't do this on the Silesia corpus compression benchmark, even though it would make sense there as well: http://mattmahoney.net/dc/silesia.html So a compressor of a few gigabyte would make sense if you would have a set of pictures of more then a few gigabyte. It's a bit similar to preprocessing text compression with a dictionary an…
Comment linking /dc/silesia.html
on “Stable Diffusion based image compression”Submitted ZPAQ: Incremental Journaling Backup Utility and Archiver as a story · 19 points
HN discussionSpecifically 1.4 "Compression is an Artificial Intelligence Problem" https://www.mattmahoney.net/dc/dce.html#Section_14
Comment linking /dc/dce.html
on “Markov chains are the original language models”Submitted Data Compression Explained as a story · 13 points
HN discussionRelated and likely inspired by the related thought: https://www.mattmahoney.net/dc/
Comment linking /dc/
on “Markov chains are the original language models”Submitted Data Compression Explained as a story · 12 points
HN discussionAre you including page history? Are you allowed to re-distribute without page history? Also, you probably know this but there is a prize for compressing bits of Wikipedia. Here's the wikipedia page about it: https://en.wikipedia.org/wiki/Hutter_Prize http://prize.hutter1.net/ About the test data: http://mattmahoney.net/dc/textdata.html And discussion on HN: https://news.ycombinator.com/item?id=7405129
Comment linking /dc/textdata.html
on “Vital Wikipedia Articles”Submitted Data Compression Explained by Matt Mahoney as a story · 12 points
HN discussion>No, xz (on a particular level setting) is both faster than bzip2 and provides better compression ratio, but zpaq is just slower. Are you implying that xz out-compresses zpaq? Can you supply a benchmark? Here's one from me - http://mattmahoney.net/dc/text.html - showing a very significant compression ratio advantage to zpaq.
Comment linking /dc/text.html
on “Stop using gzip”Submitted Data Compression Explained as a story · 6 points
HN discussionNot sure why this is being posted today or getting comments. bzip3 not a terribly impressive or useful compression algo. benchmark on enwik: https://www.mattmahoney.net/dc/text.html#1703
Comment linking /dc/text.html
on “bzip3”Submitted ZPAQ – Incremental Journaling Backup Utility and Archiver as a story · 5 points
HN discussionPreviously: “Hi, tool author here.” A useful explanation of Burrows-Wheelers transform as used by bzip3: https://news.ycombinator.com/item?id=42902407 “bzip3 is not yet listed on the large text compression benchmark” It is now: https://mattmahoney.net/dc/text.html (2 years ago, 176 comments) https://news.ycombinator.com/item?id=42899713 (4 years ago, 104 comments) https://news.ycombinator.com/item?id=31324439
Comment linking /dc/text.html
on “bzip3”Submitted Large Text Compression Benchmark as a story · 4 points
HN discussionWhen people claim that there is such a thing as "X% accuracy in reasoning", it's really hard to take anything else seriously, no matter how impressive. AI (and humans!) aside, claiming that there was an oracle that could "answer all questions" is a solved problem. Such a thing cannot exist. But this is going already too deep IMO. When people start talking about percentages or benchmark scores, there has to be some d…
Comment linking /dc/text.html
on “Study identifies weaknesses in how AI systems are evaluated”Submitted New Hutter Prize Record as a story · 4 points
HN discussionHutter prize does have speed restrictions. If it did not, LLMs would win even with counting the size of the model (which is the most reasonable choice imo) as per the main benchmark: https://www.mattmahoney.net/dc/text.html .
Comment linking /dc/text.html
on “Can gzip be a language model?”Submitted Large Text Compression Benchmark as a story · 4 points
HN discussion> The purpose of AGI is twofold: first to improve the efficiency of organizations of humans by facilitating communication and access to information, and second, to automate the economy with respect to those functions that require human labor. > With respect to communication, the goal is to make information easy to find and publish. ... When enough narrowly intelligent experts are added to the network, you should not…
Comment linking /agi2.html
on “AI startup claims to automate app making but actually just uses humans”Submitted Rationale for a Large Text Compression Benchmark (2009) as a story · 4 points
HN discussionThis is very interesting but for people who are not from the field note 1. The compressors he is comparing to *are* neural networks themselves (PAQ) 2. He is using a rather small corpus enwik8 (100,000,000 bytes), instead of the common enwik9 3. He is not publishing the amount of RAM or hardware he used http://mattmahoney.net/dc/dce.html (seems down) https://web.archive.org/web/20210216231934/http://mattmahone... It…
Comment linking /dc/dce.html
on “NNCP: Lossless Data Compression with Neural Networks (2019)”Submitted Large Text Compression Benchmark as a story · 3 points
HN discussion
