Top all-time · #4695
Jay Mody
Personal blog · jaykmody.com/blogaiprogramming
Jay Mody's personal blog at jaykmody.com covers ai and programming. Since 2023, 17 different Hacker News readers have linked it 22 times, most often “GPT in 60 Lines of NumPy”. The last post was in 2023, but the archive is still widely read.
Too few sampled comments (18) to say how often HN recommends it.?
Why HN links it
HN readers link Jay Mody's posts to learn transformers by implementing GPT-2 in about 60 lines of NumPy, plus code-first explanations of attention and diffusion models.
Summarised by Claude from a sample of 18 HN comments that link here. Each comment is marked recommending, critical or neutral (citing without judging); how this works.
A blog about things and stuff.
Most-linked posts?
- GPT in 60 Lines of NumPy11 people · 16 mentions · first linked 2023-02-09
- Speculative Sampling on LLM2 people · 2 mentions · first linked 2023-12-04
- An Intuition for Attention (from the seminal Attention Is All You Need Paper)2 people · 2 mentions · first linked 2023-04-30
What Hacker News says?
And if you want to understand I'd recommend this post (gpt2 in 60 lines of numpy) and the post on attention it links to. The concepts are mostly identical to llama, just with a few minor architectural tweaks. https://jaykmody.com/blog/gpt-from-scratch/
Comment linking /blog/gpt-from-scratch/
on “Llama3 implemented from scratch”Submitted GPT in 60 Lines of NumPy as a story · 3 points
HN discussionThis is excellent. Thanks for sharing. It's always good to go back to the fundamentals. There's another resource that is also quite good: https://jaykmody.com/blog/gpt-from-scratch/
Comment linking /blog/gpt-from-scratch/
on “Building LLMs from the Ground Up: A 3-Hour Coding Workshop”Submitted An Intuition for Attention (from the seminal Attention Is All You Need Paper) as a story · 2 points
HN discussionhttps://jaykmody.com/blog/gpt-from-scratch/ for a gpt2 inference engine in numpy then https://www.dipkumar.dev/becoming-the-unbeatable/posts/gpt-k... for adding a kv cache implementation
Comment linking /blog/gpt-from-scratch/
on “Implementing a ChatGPT-like LLM from scratch, step by step”Submitted Speculative Sampling as a story · 1 points
HN discussion> Uh oh! We’re getting NaNs! It seems our values are too high, and when being passed to the next encoder, they end up being too high and exploding! This is called gradient explosion. As far as I understand this is wrong. You're not computing gradients at any point, so this is no gradient explosion. I believe the problem is with the implementation of softmax, here [0] you have an explanation of how to implement a num…
Comment linking /blog/stable-softmax/
on “Understand how transformers work by demystifying the math behind them”Submitted Speculative Sampling on LLM as a story · 1 points
HN discussionoh this has code! great stuff. diffusion papers are famous for a lot of equations ( https://twitter.com/cto_junior/status/1766518604395155830 ) but code is much more legible (and precise?) for the rest of us. all theory papers should come with reference impl code. i'd love an extension of this for the diffusion transformer, which drives Sora and other videogen models. maybe combine this post with https://jaykmody.co…
Comment linking /blog/gpt-from-scratch/
on “Diffusion models from scratch, from a new theoretical perspective”I did this [0] (gpt in rust) with picogpt, following the great blog by jaykmody [1]. [0]: https://github.com/enricozb/picogpt-rust [1]: https://jaykmody.com/blog/gpt-from-scratch/
Comment linking /blog/gpt-from-scratch/
on “RustGPT: A pure-Rust transformer LLM built from scratch”https://jaykmody.com
Comment linking /
on “Ask HN: Could you share your personal website here?”In that case, you might also enjoy https://jaykmody.com/blog/gpt-from-scratch/ (here's the raw code: https://github.com/jaymody/picoGPT/blob/main/gpt2.py )
Comment linking /blog/gpt-from-scratch/
on “Understanding Automatic Differentiation in 30 lines of Python”Here are some high level answers: "7B" refers to the number of parameters or weights for a model. For a specific model, the versions with more parameters take more compute power to train and perform better. A foundational model is the part of a ML model that is "pretrained" on a massive data set (and usually is the bulk of the compute cost). This is usually considered the "raw" model after which it is fine-tuned for…
Comment linking /blog/gpt-from-scratch/
on “XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens”The transformer figures out that wants and cash are both verbs (both words can also be nouns). We’ve represented this added context as red text in parentheses, but in reality, the model would store it by modifying the word vectors in ways that are difficult for humans to interpret. These new vectors, known as a hidden state, are passed to the next transformer in the stack. This isn't really true, it's like when we s…
Comment linking /blog/gpt-from-scratch/
on “A jargon-free explanation of how AI large language models work”[1] is a good start, although if you want to train from scratch on CPU, you'll have to downscale as transformers need quite a bit of data before they learn to use position embeddings. For example try a single-layer RNN on Shakespeare texts [2] or a list of movie titles from IMDB [3]. You'll have to fill in the blanks because things have evolved quite a bit since those were used for language models, but you can find…
Comment linking /blog/gpt-from-scratch/
on “MIT Course: Generative AI for Constructive Communication”I wrote an article that attempts to explain attention in a more intuitive manner, without any PyTorch or other deep learning specific context: https://jaykmody.com/blog/attention-intuition/
Comment linking /blog/attention-intuition/
on “Revealing example of self-attention, the building block of transformer AI models”The more of these the better! I find when I'm learning something new I end up wanting to see a bunch of different versions of the same exposition until one sticks. I also recommend this one for following along in code: https://jaykmody.com/blog/gpt-from-scratch/
Comment linking /blog/gpt-from-scratch/
on “The transformer model, explained clearly”Some other good resources: [0]: The original paper: https://arxiv.org/abs/1706.03762 [1]: Full walkthrough for building a GPT from Scratch: https://www.youtube.com/watch?v=kCc8FmEb1nY [2]: A simple inference only implementation in just NumPy, that's only 60 lines: https://jaykmody.com/blog/gpt-from-scratch/ [3]: Some great visualizations and high-level explanations: http://jalammar.github.io/illustrated-transformer/…
Comment linking /blog/gpt-from-scratch/
on “What Is a Transformer Model?”https://jaykmody.com/blog/gpt-from-scratch/
Comment linking /blog/gpt-from-scratch/
on “Ask HN: Tutorial on LLM / already grasp neural nets”Transformer inference is ~60 lines of numpy[0]. It would be nice to just have this and not all of pytorch and transformers. [0] https://jaykmody.com/blog/gpt-from-scratch/
Comment linking /blog/gpt-from-scratch/
on “VectorDB: Vector Database Built by Kagi Search”I'd nominate Georgi Gerganov, Andrej Karpathy and Jay Mody, for https://github.com/ggerganov/llama.cpp/tree/master https://github.com/karpathy/llama2.c and https://jaykmody.com/blog/gpt-from-scratch/ respectively. These three projects cut through the crap and make llms something you can code up and run on your own, and let you bypass the (in my opinion way too abstract) mainstream pytorch/HF llm ecosystem.
Comment linking /blog/gpt-from-scratch/
on “AI and Open Source in 2023”I found this extremely helpful even though it's not in C++: https://jaykmody.com/blog/gpt-from-scratch/ .
Comment linking /blog/gpt-from-scratch/
on “C++ GPT-2 inference engine”
