Top all-time · #3900
Simon Boehm
Personal blog · siboehm.com/articles · on HN as siboehmprogrammingaisystems
Simon Boehm writes siboehm, a personal blog at siboehm.com about programming, ai and systems. Since 2021, 20 different Hacker News readers have linked it 23 times, most often “How to optimize a CUDA matmul kernel for cuBLAS-like performance (2022)”. The last post was in 2023, but the archive is still widely read.
Too few sampled comments (11) to say how often HN recommends it.?
Why HN links it
HN readers link this blog for its CUDA matrix multiplication optimization walkthroughs, which show how far a naive kernel is from a well-tuned one. Commenters use it as a go-to introduction to GPU kernel writing.
Summarised by Claude from a sample of 11 HN comments that link here. Each comment is marked recommending, critical or neutral (citing without judging); how this works.
Simon Boehm's technical blog
Most-linked posts?
- How to optimize a CUDA matmul kernel for cuBLAS-like performance (2022)13 people · 14 mentions · first linked 2023-01-05
- Pipeline Parallelism: Distributed Training via Model Partitioning2 people · 2 mentions · first linked 2024-01-17
- Fast Multidimensional Matrix Multiplication on CPU from Scratch (2022)2 people · 2 mentions · first linked 2023-08-25
What Hacker News says?
I know a trick, although it relies on the closed source cuBLAS library. The matrix multiplication in non-batched forward passes is really matrix-vector multiplication, but you can shoehorn that into matrix-matrix multiplication by setting the right dimension to 1. cuBLAS uses column major ordering to match Fortran instead of the row major ordering of C/C++, although you can swap transa and transb, m and n, a and b,…
Comment linking /articles/22/CUDA-MMM
on “Fast LLM Inference From Scratch (using CUDA)”Submitted How to optimize a CUDA matmul kernel for cuBLAS-like performance (2022) as a story · 103 points
HN discussionModal's GPU glossary is a good overview about how GPUs work [0]. Karpathy's LLM overview is a good high level overview on LLMs [1]. 3b1b's video (and subsequent videos) on transformers was excellent at helping me understand the math at a high level [2]. This matrix multiplication optimization worklog helped me understand writing better CUDA (not for beginner intro though) [3]. During this process I also asked ChatGP…
Comment linking /articles/22/CUDA-MMM
on “Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken”Submitted Fast Multidimensional Matrix Multiplication on CPU from Scratch (2022) as a story · 74 points
HN discussionHave you tried to write a kernel for basic matrix multiplication? Because I have and I can assure you it is very hard to get 50% of maximum FLOPs, let alone 90%. It is nothing like CPUs where you write a * b in C and get 99% of the performance by the compiler. Here is an example of how hard it is: https://siboehm.com/articles/22/CUDA-MMM And this is just basic matrix mult. If you add activation functions it will slo…
Comment linking /articles/22/CUDA-MMM
on “Why is Japan still investing in custom floating point accelerators?”Submitted Lleaves – Compiling decision trees for fast prediction using LLVM as a story · 4 points
HN discussionIf you ever feel like improving your matmul kernel you can check this blog https://siboehm.com/articles/22/CUDA-MMM
Comment linking /articles/22/CUDA-MMM
on “Micrograd-CUDA: adapting Karpathy's tiny autodiff engine for GPU acceleration”Submitted Becoming a Better Programmer by Tightening Feedback Loops as a story · 2 points
HN discussionFor a very deep dive into the subject, this is a great writeup: How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance ( https://siboehm.com/articles/22/CUDA-MMM ) (It's CUDA-specific, so there may be aspects that can't yet be ported to WGPU)
Comment linking /articles/22/CUDA-MMM
on “Optimizing a WebGPU Matmul Kernel for 1 TFLOP”Submitted Pipeline Parallelism: Distributed Training via Model Partitioning as a story · 2 points
HN discussionhttps://siboehm.com/articles/22/CUDA-MMM is a good start.
Comment linking /articles/22/CUDA-MMM
on “Initial CUDA Performance Lessons”Submitted How to Optimize a CUDA Matmul Kernel for cuBLAS-Like Performance: A Worklog as a story · 1 points
HN discussionThis is a really good post, and a nice successor to the earlier one that I've pointed people at before: https://siboehm.com/articles/22/CUDA-MMM (this is linked in the post, of course). That one is a good introduction but I found that it doesn't really explain a lot of the choices it makes beyond the basic ones (coalescing=good, bank conflicts=bad). This is makes it a good introduction but this goes a lot further in…
Comment linking /articles/22/CUDA-MMM
on “How to Write a Fast Matrix Multiplication from Scratch with Tensor Cores (2024)”Submitted René Girard and Mimetic Theory for Non-Philosophers as a story · 1 points
HN discussionMany of the optimizations here rely heavily on the size of matrix and it's relationship to hardware specific details, like LDS size, how they're banked and register count. It's probably not surprising that you can grind a decent improvement over a general solution, and many of the improvements shown here will need to be re-balanced, or even simply not work, for kernels working on different matrix layouts. Similarly…
Comment linking /articles/22/CUDA-MMM
on “Optimizing Matrix Multiplication on RDNA3”Submitted Pipeline-Parallelism: Distributed Training via Model Partitioning as a story · 1 points
HN discussionI've been running a setup with Recoll and https://github.com/ArchiveBox/ArchiveBox for a few months now [1]. Each morning archivebox scrapes all new links that I've put into my (text-based) notes and saves them as HTML singlefiles. Then Recoll indexes them. It's very fast and ~4 lines of code. It's surprising how often I rediscover old blog posts & papers that are much better than what Google yields me. From my expe…
Comment linking /articles/21/a-local-search-engine
on “Recoll: A desktop full-text search tool”Submitted Fast Multidimensional Matrix Multiplication on CPU from Scratch as a story · 1 points
HN discussionThis is essentially the same as https://siboehm.com/articles/22/CUDA-MMM I wonder if the same person wrote it.
Comment linking /articles/22/CUDA-MMM
on “Writing high-performance matrix multiplication kernels for Blackwell”Submitted How to Optimize a CUDA Matmul Kernel for CuBLAS-Like Performance: A Worklog as a story · 1 points
HN discussionThe code in this article is incorrect. The CUDA kernel is never called: https://github.com/RijulTP/GPUToolkit/blob/f17fec12e008d0d37... I'd also like to point out that 90 % of the time spent to "compute" the Mandelbrot set with the JIT-compiled code is spent on compiling the function, not on computation. If you actually want to learn something about CUDA, implementing matrix multiplication is a great exercise. Here…
Comment linking /articles/22/CUDA-MMM
on “GPU Survival Toolkit for the AI age: The bare minimum every developer must know”Submitted Data-parallel distributed training of deep learning models as a story · 1 points
HN discussionI keep this link in my favorites and refer to it every now and again. Still one of the best write-ups I've seen on just have vast the difference is between a naive and well tuned kernel https://siboehm.com/articles/22/CUDA-MMM
Comment linking /articles/22/CUDA-MMM
on “Training an LLM in Swift, Part 1: Taking matrix mult from Gflop/s to Tflop/s”
