Top all-time · #3039
Jascha’s blog
Personal blog · sohl-dickstein.github.ioaiscience
Jascha’s blog is a personal blog at sohl-dickstein.github.io about ai and science. Since 2022, 25 different Hacker News readers have linked it 27 times, most often “Too much efficiency makes everything worse (2022)”. The last post was in 2024, but the archive is still widely read.
Too few sampled comments (15) to say how often HN recommends it.?
Why HN links it
HN readers link this blog's post on 'strong Goodhart' and overfitting, which argues that optimizing a proxy metric undermines the real goal, and its post on fractal boundaries in neural network training.
Summarised by Claude from a sample of 15 HN comments that link here. Each comment is marked recommending, critical or neutral (citing without judging); how this works.
This blog is intended to be a place to share ideas and results that are too weird, incomplete, or off-topic to turn into an academic paper, but that I think may be important. Let me know what you think! Contact links to the left.
Most-linked posts?
- Too much efficiency makes everything worse (2022)12 people · 14 mentions · first linked 2022-11-11
- AI misalignment: intelligent agents behave less coherently6 people · 6 mentions · first linked 2023-03-09
- Neural network training makes beautiful fractals5 people · 5 mentions · first linked 2024-02-12
What Hacker News says?
This reminds me of something that appeared on hacker news a couple days ago: https://sohl-dickstein.github.io/2022/11/06/strong-Goodhart.... Basically, the proxy objective is "getting into YC" -- the real objective is producing value of course. We've started optimizing for the proxy. Which produces outcomes like this, where YC is a Thing You Do, after Graduating From Stanford, and before Becoming a YC Alum.
Comment linking /2022/11/06/strong-Goodhart.html
on “Y Combinator Traded Prestige for Growth”Submitted Too much efficiency makes everything worse (2022) as a story · 917 points
HN discussionTest scores do not measure directly what they purport to measure. They are a proxy, and when you have a system designed to optimize for a proxy of the thing you want to improve, then the system will always find ways to exploit the difference between the proxy and the underlying thing. You can call it "juking the stats" or "overfitting", or whatever. https://sohl-dickstein.github.io/2022/11/06/strong-Goodhart....
Comment linking /2022/11/06/strong-Goodhart.html
on “US High school students' scores fall in reading and math”Submitted Neural network training makes beautiful fractals as a story · 316 points
HN discussionPoint taken about the level of discourse on the internet and the mainstream media, no argument there. It is painful, but just because the discourse around the topic is smooth brained doesn't mean that the position is wrong. This is something that I struggle with. Sometimes the discourse is so unimaginably stupid that it pushes you to default to the other side's point of view, but that doesn't actually mean you got t…
Comment linking /2022/11/06/strong-Goodhart.html
on “US antitrust case against Amazon to move forward”Submitted Overfitting and the strong version of Goodhart’s law as a story · 187 points
HN discussionHere's the associated blog post, which includes the videos: https://sohl-dickstein.github.io/2024/02/12/fractal.html Not a ML'er so not sure what to make of it, beyond a fascinating connection.
Comment linking /2024/02/12/fractal.html
on “Neural network training makes beautiful fractals”Submitted Neural network training makes beautiful fractals as a story · 6 points
HN discussionThe phenomenon being discussed here is a type of overfitting: https://sohl-dickstein.github.io/2022/11/06/strong-Goodhart.... The last 50 years or so of managerial practice has been a recipe for overfitting with a brutal emphasis on measuring, optimizing, and stack ranking everything. I think an argument can be made that this is an age of overfitting everywhere.
Comment linking /2022/11/06/strong-Goodhart.html
on “Are people bad at their jobs or are the jobs just bad?”Submitted Too much efficiency makes everything worse as a story · 3 points
HN discussionOne of my favourite suggestions to counter this from an essay on overfitting I really like [1] is that applicant's probability of being accepted is proportional to their position in the ranking. The downside is that some great applicants will not enter their top choice of school, and some people who aren't great fits will. But, on the other hand, the perverse incentive of spending the entirety of high-school optimis…
Comment linking /2022/11/06/strong-Goodhart.html
on “June Huh dropped out to become a poet, now he’s won a Fields Medal (2022)”Submitted Too much efficiency makes everything worse: overfitting and Goodhart's law as a story · 2 points
HN discussionThe blog post would be a better link for this submission. https://sohl-dickstein.github.io/2024/02/12/fractal.html
Comment linking /2024/02/12/fractal.html
on “The boundary of neural network trainability is fractal”Submitted AI misalignment: intelligent agents behave less coherently as a story · 2 points
HN discussionReminded me of this recent observation of fractals in nn training https://sohl-dickstein.github.io/2024/02/12/fractal.html
Comment linking /2024/02/12/fractal.html
on “'Entropy bagels' and other complex structures emerge from simple rules”Submitted The hot mess theory of AI misalignment as a story · 2 points
HN discussionThere is a great blog post [0] that sohl dickstein had a couple years ago, about what happens when the metric being optimized is only a proxy for the thing you want to maximize — what happens eventually is “overfitting”. The original metric (which we cannot directly optimize for) actually starts to get worse. I believe publishing papers and citations are those proxy metrics. [0] https://sohl-dickstein.github.io/2022…
Comment linking /2022/11/06/strong-Goodhart.html
on “Incentives in Academic Research”Submitted More intelligent agents behave less coherently (2023) as a story · 1 points
HN discussionEvery time “relentless push for efficiency destroying X” comes up I should repost this: https://sohl-dickstein.github.io/2022/11/06/strong-Goodhart.... This is a well understood phenomenon.
Comment linking /2022/11/06/strong-Goodhart.html
on “Kaiser nurses say AI, workplace surveillance are making their jobs, care worse”Submitted The hot mess theory of AI misalignment (2023) as a story · 1 points
HN discussionLove this framing I like it because it makes it clear that the boundaries of those stable structures are what encode the world model that encompasses the future. Conscious systems just decode the boundaries of these recursive structures. Corollary: LLMs aren't smart, they're just recursive structures (differently recursive than our minds) that can decode the boundary we store in language the recursion of language...…
Comment linking /2024/02/12/fractal.html
on “Mathematicians Crack a Fractal Conjecture on Chaos”Submitted Hot mess theory AI misalignment: More intelligent agents behave less coherently as a story · 1 points
HN discussionIt's nice seeing this with Sohl-Dickstein as the last author after reading this blog post from him some time ago: https://sohl-dickstein.github.io/2023/03/09/coherence.html
Comment linking /2023/03/09/coherence.html
on “The Hot Mess of AI”Submitted Brain dump on the diversity of AI risk as a story · 1 points
HN discussionI think the charitable interpretation of this article is as a case of overfitting on the proxy metric [1]. If you care exclusively about reducing CO2 emissions, then her arguments have a point. But that's not how it works in reality: Even in environmental science, CO2 emissions are only one problem, next to other greenhouse gases, ocean acidification, biodiversity loss, etc. Optimizing for CO2 emissions at the expen…
Comment linking /2022/11/06/strong-Goodhart.html
on “Environmentalists Exclusively Use the Microwave”Most of what you describe here is overfitting: https://sohl-dickstein.github.io/2022/11/06/strong-Goodhart....
Comment linking /2022/11/06/strong-Goodhart.html
on “The West Forgot How to Make Things. Now It's Forgetting How to Code”Overoptimization is the root of all evil. See also https://sohl-dickstein.github.io/2022/11/06/strong-Goodhart....
Comment linking /2022/11/06/strong-Goodhart.html
on “I got paid minimum wage to solve an impossible problem”
