Top all-time · #25189
Stephen Panaro
Personal blog · stephenpanaro.com/blog · on HN as smpanaroaiprogramming
Stephen Panaro's personal blog at stephenpanaro.com covers ai and programming. Since 2024, 3 different Hacker News readers have linked it 4 times. It is still writing: the last post was on 2025-01-14, and you can follow it by feed.
Too few sampled comments (1) to say how often HN recommends it.?
Why HN links it
HN readers link this blog for a write-up on running ModernBERT on Apple's Neural Engine, with the code included. The commenter wishes the author wrote more.
Summarised by Claude from a sample of 1 HN comments that link here. Each comment is marked recommending, critical or neutral (citing without judging); how this works.
What Hacker News says?
coremltools is the only way to run on ANE, so less of a trick and more of a requirement. The tricks are more around optimizing for the hardware capabilities/constraints. For instance: - conv2d is faster than linear (see Apple's post [0]) so you rewrite the model for that (example from the repo [1]) - inputs/outputs are static shapes, so KV cache requires some creativity (I wrote about that here [2]) - compute is flo…
Comment linking /blog/kv-cache-for-neural-engine
on “Run LLMs on Apple Neural Engine (ANE)”Submitted LLMs for Your iPhone: Whole-Tensor 4 Bit Quantization as a story · 2 points
HN discussionNot a public follow-up but the iOS 17 speech-to-text model has a clever approach to KV caching that works within the ANE’s constraints (fixed size inputs). I wrote about it here[0] but the gist is you can have a fixed size cache and slide it in chunks with each inference. Not as efficient as a cache that grows by one each time of course. [0]: https://stephenpanaro.com/blog/inside-apples-2023-transforme...
Comment linking /blog/inside-apples-2023-transformers
on “Run LLMs on Apple Neural Engine (ANE)”If anyone is interested in doing something seriously useful with these neural cores, there is this incredible write up on getting ModernBERT running on them: https://stephenpanaro.com/blog/modernbert-on-apple-neural-en... Really wish this author would blog more, this piece is incredible and includes the code. Also ModernBERT is amazing if you haven’t used it before, worth spending time with - have used it myself for…
Comment linking /blog/modernbert-on-apple-neural-engine
on “Apple Neural Engine: Architecture, Programming, and Performance”