Top all-time · #5238
152334H
Personal blog · 152334h.github.io/blogaiprogramming
152334H is a personal blog at 152334h.github.io about ai and programming. Since 2023, 15 different Hacker News readers have linked it 20 times. It is still writing: the last post was on 2026-10-13, and you can follow it by feed.
Too few sampled comments (14) to say how often HN recommends it.?
Why HN links it
HN commenters link this post on why GPT-4 and other sparse Mixture-of-Experts models give non-deterministic outputs, tied to batched inference rather than temperature. It is cited whenever LLM determinism comes up.
Summarised by Claude from a sample of 14 HN comments that link here. Each comment is marked recommending, critical or neutral (citing without judging); how this works.
152334H Personal Blog
Most-linked posts?
- /blog/non-determinism-in-gpt-4/14 people · 15 mentions · first linked 2023-08-10
What Hacker News says?
The samples your input is batched with on the provider's backend vary between calls and sparse mixture of experts routing when implemented for efficient utilization induces competition among tokens with either encouraged or enforced balance of expert usage among tokens in the same fixed-size group. I think it's unknown or at least undisclosed exactly why sequence non-determinism at zero temperature occurs in these p…
Comment linking /blog/non-determinism-in-gpt-4/
on “Ingesting PDFs and why Gemini 2.0 changes everything”Submitted Calculating the cost of a Google DeepMind paper as a story · 303 points
HN discussionFrom the good old days: https://152334h.github.io/blog/non-determinism-in-gpt-4/ (that's been a short two years). Unless something has fundamentally changed since then (which I've not heard about) all sparse models are only deterministic at the batch level, rather than the sample level.
Comment linking /blog/non-determinism-in-gpt-4/
on “Study mode”Submitted Non-determinism in GPT-4 is caused by Sparse MoE as a story · 20 points
HN discussionFor Mixture of Expert models (like GPTs are), they can produce different results for an input sequence if that sequence is retried together with a different set of sequences in its inference batch, because of the model (“expert”) routing depends on the batch, not the single sequence: https://152334h.github.io/blog/non-determinism-in-gpt-4/ And in general, binary floating point arithmetic cannot guarantee associativi…
Comment linking /blog/non-determinism-in-gpt-4/
on “Does current AI represent a dead end?”Submitted Can AI agents write kernel exploits? as a story · 1 points
HN discussionNo, I meant that the responses will be different run-to-run. [0] [0] https://152334h.github.io/blog/non-determinism-in-gpt-4/
Comment linking /blog/non-determinism-in-gpt-4/
on “Why DeepSeek is cheap at scale but expensive to run locally”Submitted Knowing Enough About MoE to Explain Dropped Tokens in GPT-4 as a story · 1 points
HN discussion>compilers behaving in a deterministic and predictable way is an important fundamental of pipelines. LLMs are inherently unpredictable, and so using an LLM for compilation / decompilation -- even an LLM that has 99.99% accuracy You're confusing different concepts here. An llm is technically not unpredictable by itself (at least the ones we are talking about here, there are different problems with beasts like GPT4 [1…
Comment linking /blog/non-determinism-in-gpt-4/
on “Meta LLM Compiler: neural optimizer and disassembler”Submitted Why can't TorToiSe be fine-tuned? as a story · 1 points
HN discussionOne theory is it is caused by its Sparse MoE (Mixture of Experts) architecture [1]: > The GPT-4 API is hosted with a backend that does batched inference. Although some of the randomness may be explained by other factors, the vast majority of non-determinism in the API is explainable by its Sparse MoE architecture failing to enforce per-sequence determinism. [1] https://152334h.github.io/blog/non-determinism-in-gpt-4/
Comment linking /blog/non-determinism-in-gpt-4/
on “Show HN: LLMs can generate valid JSON 100% of the time”It can happen due to a number of reasons, but in the case of GPT-4 it's probably because of their MoE implementation https://152334h.github.io/blog/non-determinism-in-gpt-4/
Comment linking /blog/non-determinism-in-gpt-4/
on “AI leaderboards are no longer useful. It's time to switch to Pareto curves”https://152334h.github.io/blog/non-determinism-in-gpt-4/
Comment linking /blog/non-determinism-in-gpt-4/
on “Show HN: I made a "programming language" looking for feedback”Nope. You have not shown how a large scale collection of neural networks irrespective of their architecture is more deterministic when compared to a 'compiler' and only repeating a known misconception of tweaking the temperature to 0 which does not bring the determinism you claim it brings with LLMs [0] [1] [2], otherwise you would not have this problem in the first place. By even doing that, the result of the outpu…
Comment linking /blog/non-determinism-in-gpt-4/
on “LLMs could be, but shouldn't be compilers”That's what I meant by "if you're using some API you have no control over, anything could happen." OpenAI could offer access to a fully deterministic version at a higher price point, but they choose not to and there's nothing you can do about that. FYI, your source is medium-walled blog spam of https://152334h.github.io/blog/non-determinism-in-gpt-4/
Comment linking /blog/non-determinism-in-gpt-4/
on “Advanced NLP with SpaCy”It is not deterministic since text-davinci-001. https://152334h.github.io/blog/non-determinism-in-gpt-4/
Comment linking /blog/non-determinism-in-gpt-4/
on “A GPT-4 capability forecasting challenge”A lot of stuff behind the scenes is going on to batch and route queries to GPT-4 models that are in perturbed states already[1]. This isn't gross negligence, this is basic capitalism. If you want sole access to a GPT-4 MoE cluster starting fresh, it's gonna cost you. 1. https://152334h.github.io/blog/non-determinism-in-gpt-4/
Comment linking /blog/non-determinism-in-gpt-4/
on “After 118 days, GPT Unicorn has produced a unicorn”Your incorrect assumption here is that determinism comes with no tradeoffs. There are a few outsider analyses on the topic, for example, this one on sparse MoEs [1]. If OpenAI uses sparse MoE as described, then determinism would be possible but inefficient. Even if it's not sparse MoE, chances are high that the non-determinism is introduced somewhere purely as a performance optimization. The article speculates that…
Comment linking /blog/non-determinism-in-gpt-4/
on “After 118 days, GPT Unicorn has produced a unicorn”>Whereas with a LLM if we control the seed we can 100% reproduce the same result No, you can't. For the latest GPT models and the way they are run, this doesn't work anymore, making the experiment completely illogical. Some of the reasons are explained here pretty well: https://152334h.github.io/blog/non-determinism-in-gpt-4/
Comment linking /blog/non-determinism-in-gpt-4/
on “After 118 days, GPT Unicorn has produced a unicorn”I'm not aware of anything concrete by OpenAI, but others have offered possible explanations. One idea is that the cause is batched inference in sparse MoE (mixture of experts) models. https://152334h.github.io/blog/non-determinism-in-gpt-4/ HN discussion: https://news.ycombinator.com/item?id=37006224
Comment linking /blog/non-determinism-in-gpt-4/
on “After 118 days, GPT Unicorn has produced a unicorn”
