Top all-time · #1157
Oliver Knill
Personal blog · people.math.harvard.edumathscience
Oliver Knill writes Harvard Mathematics Department : Home page, a personal blog at people.math.harvard.edu about math and science. Since 2020, 62 different Hacker News readers have linked it 71 times, most often “A Mathematical Theory of Communication [pdf]”. No feed was found, so read it on the site itself.
Most-linked posts?
- A Mathematical Theory of Communication [pdf]28 people · 33 mentions · first linked 2021-10-09
- A Mathematical Theory of Communication (1948) [pdf]11 people · 12 mentions · first linked 2020-04-27
- Mathematics in Movies3 people · 3 mentions · first linked 2022-04-18
- /~knill/teaching/mathe320_2015_fall/blog15/surreal1.pdf2 people · 2 mentions · first linked 2024-12-16
What Hacker News says?
>> Looking at early telegraphs doesn’t predict the iPhone, etc. The problem with this line of argument is that LLMs are not new technology, rather they are the latest evolution of statistical language modelling, a technology that we've had at least since Shannon's time [1]. We are way, way past the telegraph era, and well into the age of large telephony switches handling millions of calls a second. Does that mean we…
Comment linking /~ctm/home/text/others/shannon/entropy/entropy.pdf
on “I Am Tired of AI”Submitted A Mathematical Theory of Communication [pdf] as a story · 298 points
HN discussionI see there's no mention yet in this thread of A Mathematical Theory of Communication [1]. Claude Shannon launched the entire field of information theory with this paper. Admittedly, not useful to everyone , but information theory is surprisingly useful outside of the simple communications context. [1] http://people.math.harvard.edu/~ctm/home/text/others/shannon...
Comment linking /~ctm/home/text/others/shannon/entropy/entropy.pdf
on “Papers every developer should read at least twice (2009)”Submitted Some Fundamental Theorems in Mathematics (2019) [pdf] as a story · 234 points
HN discussionIn my opinion, saying entropy is a measure of randomness is confusing at best and wrong at worst. Entropy is a the amount of information it takes to describe a system. That is, how many bits does it take to "encode" all possible states of the system. For example, say I had to communicate the result of 100 (fair) coin flips to you. This requires 100 bits of information as each of the 100 bit vectors is equally likely…
Comment linking /~ctm/home/text/others/shannon/entropy/entropy.pdf
on “Ask HN: What scientific phenomenon do you wish someone would explain better?”Submitted A Mathematical Theory of Communication (1948) [pdf] as a story · 197 points
HN discussion> the topics in the parent post should not be a major surprise to anyone who has read https://people.math.harvard.edu/~ctm/home/text/others/shanno ... ! > which clearly explains it (and said emergent phenomena) Very smart information theory people have looked at neural networks through the lens of information theory and published famous papers about it years ago. It couldn't explain many things about neural networks…
Comment linking /~ctm/home/text/others/shanno
on “Beyond self-attention: How a small language model predicts the next token”Submitted The Wobbly Table Theorem (2022) as a story · 45 points
HN discussionSome of the topics in the parent post should not be a major surprise to anyone who has read https://people.math.harvard.edu/~ctm/home/text/others/shanno... ! If we do not have read the foundations of the field that we are in, we are doomed to be mystified by unexplained phenomena which arise pretty naturally as consequences of already-distilled work! That said, the experiments seem very thorough, on a first pass/ini…
Comment linking /~ctm/home/text/others/shannon/entropy/entropy.pdf
on “Beyond self-attention: How a small language model predicts the next token”Submitted A Mathematical Theory of Communication [pdf] as a story · 5 points
HN discussionI think you would enjoy reading Claude Shannon's seminal paper on information theory. What you are proposing is profoundly silly. But to understand why it's profoundly silly is very much worthwhile! http://people.math.harvard.edu/~ctm/home/text/others/shannon...
Comment linking /~ctm/home/text/others/shannon/entropy/entropy.pdf
on “Bitcoin Is Time”Submitted A Mathematical Theory of Communication, Claude Shannon [pdf] as a story · 5 points
HN discussionGravity's Rainbow is downright abstract at times, but if you use a guide like this [0] then you can move beyond trying to figure out what actually happens and really enjoy how Pynchon twists language and sentences into incredible images and scenes. Some of it is, for lack of a better term, downright fantastical rather than literal. [0]: https://people.math.harvard.edu/%7Ectm/links/culture/rainbow...
Comment linking /%7Ectm/links/culture/rainbow.bell.html
on “The joy of reading books you don't understand”Submitted Some fundamental theorems in math – Oliver KNILL [pdf] as a story · 3 points
HN discussionRadix economy is all about which base is the most efficient to represent a given number. It is simple to show that, for large numbers, this is equivalent to how efficient a base can represent itself, b/ln(b). Simple calculus shows this is minimized at e (Euler's number) or 3 if integer (closely followed by 2). It sounds like you have something to add but you are already dictating the base by saying "bit". Literally…
Comment linking /~ctm/home/text/others/shannon/entropy/entropy.pdf
on “Towards 1-bit Machine Learning Models”Submitted Mathematics in Movies as a story · 3 points
HN discussionGood luck with that EU, it's dead on arrival. The day a politician tells a mathematician what to think is the day a mathematician instructs a politician on how to think. Real criminals have good opsec, it's part of the job. Non-criminals want provable privacy and security. Deluded and ignorant politicians want a PR stunt with good optics. If I, as a relatively mathematical individual, employ mathematical methods of…
Comment linking /~ctm/home/text/others/shannon/entropy/entropy.pdf
on “IT companies warn in open letter: EU wants to ban encryption”Submitted A Mathematical Theory of Communication [pdf] as a story · 2 points
HN discussion> the typical course is still not that much different 15 years later. For context, I checked your profile to see where you did your undergraduate degree. I am familiar with the way calculus is currently taught at that university, and it looks quite similar to the "radical [...] era-appropriate" textbook that you linked (at least based on a quick read of a few chapters). Those courses are also taught in a quasi-activ…
Comment linking /~community/inclusive-classrooms.html
on “What is the inverse of a vector?”Submitted What should a professional mathematician know? [pdf] as a story · 2 points
HN discussionShannon's original paper "A Mathematical Theory of Communication" which introduced information theory is so accessible as to be useful as a tutorial itself: http://people.math.harvard.edu/~ctm/home/text/others/shannon...
Comment linking /~ctm/home/text/others/shannon/entropy/entropy.pdf
on “Information Theory: A Tutorial Introduction”Submitted Getting the facts right – the Water-Candle experiment as a story · 2 points
HN discussionHi! Entropy unfortunately much more complicated than that in practice, mainly as actually finding the real underlying entropy of a variable is quite difficult in practice! However, we can define it as a quantity with respect to different values. But the entropy of a variable as estimated by the model is generally not the actual entropy of the variable, and this gets worse for sequences -- we can maybe upper bound th…
Comment linking /~ctm/home/text/others/shannon/entropy/entropy.pdf
on “Detecting when LLMs are uncertain”Submitted Differential Geometry – Oliver Knill (2025) [pdf] as a story · 1 points
HN discussionA lot of the ML practitioners (including myself) that I know think that this is a pretty ridiculous algorithm, unfortunately. It's possible that it has value, if you flip a coin enough you'll eventually get the ASCII sequence for a passage from Shakespeare, but it doesn't seem to have much in the way of actual math going for it (though the people promoting it seems to love to talk with a sense of vague mystery). It…
Comment linking /~ctm/home/text/others/shannon/entropy/entropy.pdf
on “Detecting when LLMs are uncertain”Submitted A Mathematical Theory of Communication [pdf] as a story · 1 points
HN discussionThis is what Shannon disproved in the paper that founded information theory. It never becomes impossible, even at a fixed signal power; the noise reduces the channel capacity, but never to zero. https://people.math.harvard.edu/~ctm/home/text/others/shanno...
Comment linking /~ctm/home/text/others/shannon/entropy/entropy.pdf
on “When the physicists need burner phones, that's when you know America's changed”Submitted Mathematics in Movies as a story · 1 points
HN discussionYou might enjoy Oliver Knill’s lecture notes on undergrad math topics, which are similarly concise: http://people.math.harvard.edu/~knill/teach/index.html
Comment linking /~knill/teach/index.html
on “Some Fundamental Theorems in Mathematics (2019) [pdf]”Submitted The Mathematical Theory of Communication [pdf] as a story · 1 points
HN discussion
