Top all-time · #2505
srush.github.io
Personal blog · srush.github.ioaiprogramming
srush.github.io is a personal blog at srush.github.io about ai and programming. Since 2022, 29 different Hacker News readers have linked it 34 times, most often “The Annotated S4”. No feed was found, so read it on the site itself.
Too few sampled comments (18) to say how often HN recommends it.?
Why HN links it
HN readers link Sasha Rush's blog for explanations of state-space models, especially The Annotated S4, and for RASP, a language for programming transformers. Commenters call the S4 primer a very good primer.
Summarised by Claude from a sample of 18 HN comments that link here. Each comment is marked recommending, critical or neutral (citing without judging); how this works.
Most-linked posts?
- The Annotated S412 people · 13 mentions · first linked 2023-05-06
- Thinking Like Transformer7 people · 8 mentions · first linked 2022-12-23
- A Tour of Differentiable Rasterization3 people · 3 mentions · first linked 2024-06-17
- Lean Verified Transformers2 people · 2 mentions · first linked 2026-09-15
- Sampling to Reinforce2 people · 2 mentions · first linked 2026-10-01
- Mamba: The Hard Way2 people · 2 mentions · first linked 2024-02-19
What Hacker News says?
You can program algorithms into transformer networks, up to the limit of how many computations you get. https://srush.github.io/raspy/ Are you going to do computer reductionism too and say computers can't do arithmetic, they just run electricity through silicon?
Comment linking /raspy/
on “Visualizing Attention, a Transformer's Heart [video]”Submitted The Annotated S4 as a story · 111 points
HN discussion> Why does, 'mat' follow from 'the cat sat on the ...' You're confidently incorrect by oversimplifying all LLMs to a base model performing a completion from a trivial context of 5 words. This is tantamount to a straw man. Not only do few people use untuned base models, it completely ignores in-context learning that allows the model to build complex semantic structures from the relationships learnt from its training…
Comment linking /raspy/
on “Visualizing Attention, a Transformer's Heart [video]”Submitted Lean Verified Transformers as a story · 4 points
HN discussionA very good primer to state-space models (from which Mamba is based on) is The Annotated S4 [1]. If you want to dive into the code I wrote a minimal single-file implementation of Mamba-2 here [2]. [1]: https://srush.github.io/annotated-s4/ [2]: https://github.com/tommyip/mamba2-minimal
Comment linking /annotated-s4/
on “Codestral Mamba”Submitted A Tour of Differentiable Rasterization as a story · 3 points
HN discussionHMMs are much more similar to S3, S4, and other deep state-space models than to transformers. In fact, HMMs are discrete shallow state-space models. I believe S4 and successors might become serious contenders to transformers. For a detailed tutorial, see https://srush.github.io/annotated-s4
Comment linking /annotated-s4
on “Markov Chains are the Original Language Models”Submitted Mamba: The Hard Way as a story · 3 points
HN discussionOh hey, I wrote this. Been a long time. I had the lucky of break of working in machine translation / parsing when the most important invention of the century happened in my niche field. I'm pretty interested in the intersection of code / ML. If that's your thing here are some other writing you might be interested in. * Thinking about cuda: http://github.com/srush/gpu-puzzles * Tensors considered harmful: https://nlp…
Comment linking /annotated-s4/
on “The Annotated Transformer (2022)”Submitted A Tour of Differentiable Rasterization as a story · 2 points
HN discussionI am told that an interesting alternative is the Structured State Space for Sequence Modeling (S4). I don't personally know much about this technique, but didn't see anybody else mention this in this thread. https://srush.github.io/annotated-s4/
Comment linking /annotated-s4/
on “Ask HN: Is anybody building an alternative transformer?”Submitted Thinking Like Transformer as a story · 2 points
HN discussionThis is an article about "S4" that uses "Structured State Spaces". https://srush.github.io/annotated-s4/ The matrices that make up the state space (A, B and C) are constant in S4. This allowed them to represent some of the math operations as a convolution (which can be parallelized). The difference between S4 and Mamba is that these matrices are input-dependent in Mamba. Plus they add in some CUDA stuff ("parallel s…
Comment linking /annotated-s4/
on “MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts”Submitted A Tour of Differentiable Rasterization as a story · 2 points
HN discussionAn alternative which I've used with some succes are structured state space models: https://srush.github.io/annotated-s4/ . A very different approach that works well for quite a few types of problems.
Comment linking /annotated-s4/
on “Unlimiformer: Long-Range Transformers with Unlimited Length Input”Submitted From Sampling to Reinforce as a story · 1 points
HN discussionSee https://srush.github.io/annotated-s4 . I also think state-space models will make a comeback.
Comment linking /annotated-s4
on “Ask HN: What are your predictions for 2024?”Submitted Sampling to Reinforce as a story · 1 points
HN discussionYou can manually program transformers: https://srush.github.io/raspy/ I don't know if you can integrate them into a model. I think you might run out of space, since these aren't polysemantic and so would take up a lot more "room" than learned neurons.
Comment linking /raspy/
on “Decomposing Language Models into Understandable Components”Submitted Lean Verified Transformers as a story · 1 points
HN discussionStill chewing but my takeaways thus far: SSM models strengths are with continuous data like audio and video, they struggle with discrete data like text/ DNA. This newest architecture uses selective attention to try to address the weaknesses around discrete data with some loss in performance in continuous tasks, empirically shown here with audio. The empirical exploration was limited to smaller size models, the perfo…
Comment linking /annotated-s4/
on “Mamba outperforms transformers "everywhere we tried"”Submitted Mamba: The Hard Way as a story · 1 points
HN discussionStill chewing but my takeaways thus far: SSM models strengths are with continuous data like audio and video, they struggle with discrete data like text/ DNA. This newest architecture uses selective attention to try to address the weaknesses around discrete data with some loss in performance in continuous tasks, empirically shown here with audio. The empirical exploration was limited to smaller size models, the perfo…
Comment linking /annotated-s4/
on “Mamba: Linear-Time Sequence Modeling with Selective State Spaces”Submitted The Annotated S4 as a story · 1 points
HN discussionI got excited about this a couple years ago and ported it to Pyodide in to run in the browser with a pytorch like library: https://srush.github.io/g9py/ It's a little choppier than javascript, but crazy to me that it works at all.
Comment linking /g9py/
on “G9.js: Automatically Interactive Graphics”Submitted The Annotated S4 as a story · 1 points
HN discussionTo quote someone: RASP is like Matlab, designed by Satan. There is an interpreter for a RASP like language if you want to try it out: https://srush.github.io/raspy/ And deepmind published a compiler from RASP to Transformer weights: https://github.com/deepmind/tracr
Comment linking /raspy/
on “Thinking Like Transformers (pdf)”Submitted S4: Efficiently Modeling Long Sequences with Structured State Spaces as a story · 1 points
HN discussionRNN-RWKV - https://news.ycombinator.com/item?id=36038868 H3 - https://news.ycombinator.com/item?id=34673535 S4 - https://srush.github.io/annotated-s4/
Comment linking /annotated-s4/
on “Why the Original Transformer LLM Figure Is Wrong, and Other Interesting Tidbits”Submitted Thinking Like Transformers as a story · 1 points
HN discussion
