Top all-time · #15860
Harm de Vries
Personal blog · harmdevries.com/postaiprogramming
Harm de Vries's personal blog at harmdevries.com covers ai and programming. Since 2023, 5 different Hacker News readers have linked it 6 times, most often “We should train smaller LLMs on more tokens”. The last post was in 2023, but the archive is still widely read.
Too few sampled comments (5) to say how often HN recommends it.?
Why HN links it
HN readers link this blog's post on model size versus compute overhead to give context on LLM training scaling, the sweet spot between parameters and tokens, and inference cost trade-offs.
Summarised by Claude from a sample of 5 HN comments that link here. Each comment is marked recommending, critical or neutral (citing without judging); how this works.
AI researcher focused on LLM agents. Co-founder of BigCode, previously led the Large Language Models Lab at ServiceNow Research.
Most-linked posts?
- We should train smaller LLMs on more tokens5 people · 6 mentions · first linked 2023-04-10
What Hacker News says?
I think "sweet spot" is going to depend on your task, but here's a good recent paper that may give you some more context on thinking about training and model sizes: https://www.harmdevries.com/post/model-size-vs-compute-overh... There have also been quite a few developments on sparsity lately. Here's a technique SparseGPT which suggests that you can prune 50% of parameters with almost no loss in performance for exam…
Comment linking /post/model-size-vs-compute-overhead/
on “Stability AI Launches the First of Its StableLM Suite of Language Models”Submitted We should train smaller LLMs on more tokens as a story · 1 points
HN discussionAccording to this LLaMA still didn't go far enough: https://www.harmdevries.com/post/model-size-vs-compute-overh...
Comment linking /post/model-size-vs-compute-overhead/
on “Stability AI Launches the First of Its StableLM Suite of Language Models”Interestingly, the models themselves also have compute cost associated with them, namely the cost of running them (inference). The smaller a model is, the less compute it needs for inference. Which is an interesting trade-off, because you can overtrain (rather than undertrain, as in the past) a model, i.e. use more tokens, fewer parameters, and ultimately more compute than optimal during training, to get lower infer…
Comment linking /post/model-size-vs-compute-overhead/
on “Gradients Are Not All You Need”This is almost exactly half the tokens (or twice the model size) that the Chinchilla scaling laws would predict (in fact, Table 3 in the Chinchilla paper gives 3.7T tokens, almost exactly this number, as optimal for a 175B parameter model). Some comments on https://news.ycombinator.com/item?id=35891444 back when it was mostly assumed Palm2 was Chinchilla optimal said it was surprising that they hadn't taken inferenc…
Comment linking /post/model-size-vs-compute-overhead/
on “Palm 2 – a 340B model trained on 3.6 Trillion tokens of text”While I agree we probably aren't getting exponentially increasing parameter counts (GPT4 is by all accounts 1T paramaters and of course, it is significantly better than GPT3) we are still seeing lots of improvements - 3.5 is much better than 3, based "just" on InstructGPT/RLHF training. Models are getting better as well - LLaMA 30B beats/matches GPT-3 on raw eval benchmarks at 1/6 the parameter count. We're also see…
Comment linking /post/model-size-vs-compute-overhead/
on “GitHub Copilot emits GPL code”