Top all-time · #1922
Sebastian Ruder
Personal blog · ruder.ioaidata
Sebastian Ruder's personal blog at ruder.io covers ai and data. Since 2017, 38 different Hacker News readers have linked it 50 times, most often “NLP's ImageNet moment has arrived (2018)”. The last post was in 2024, but the archive is still widely read.
Too few sampled comments (14) to say how often HN recommends it.?
Why HN links it
HN readers link Sebastian Ruder's blog as a guide to NLP research: concise, accessible posts on recent advances, literature reviews, and deep dives into topics like word embeddings and optimizers.
- Recommended for · 3 comments
- Concise, accessible posts on recent NLP advances; deep dives into topics; a good run-down of optimizers; a word embeddings overview recommended.
Summarised by Claude from a sample of 14 HN comments that link here. Each comment is marked recommending, critical or neutral (citing without judging); how this works.
Sebastian Ruder's NLP and ML blog
Most-linked posts?
- NLP's ImageNet moment has arrived (2018)5 people · 5 mentions · first linked 2018-10-31
- An overview of gradient descent optimization algorithms (2016)2 people · 4 mentions · first linked 2023-04-10
- Word embeddings in 2017: Trends and future directions2 people · 2 mentions · first linked 2017-10-21
- A collection of interesting research directions around transfer learning and NLP2 people · 2 mentions · first linked 2018-03-04
- /nlp-imagenet/2 people · 2 mentions · first linked 2023-09-14
- An overview of gradient descent optimization algorithms2 people · 2 mentions · first linked 2017-07-09
- Do NLP Beyond English2 people · 2 mentions · first linked 2020-08-01
- A survey of cross-lingual word embedding models (2016)2 people · 2 mentions · first linked 2018-01-16
What Hacker News says?
Thanks for this reference, I will look it up. Though, from my experience people in NLP still (be default) train from scratch, with some exceptions for tasks on the same dataset: - https://blog.openai.com/unsupervised-sentiment-neuron/ - http://ruder.io/nlp-imagenet/
Comment linking /nlp-imagenet/
on “The Unreasonable Effectiveness of Deep Feature Extraction”Submitted Deep Learning for NLP Best Practices as a story · 195 points
HN discussionYou might be thinking of this article by Sebastian Ruder: https://www.ruder.io/nlp-imagenet/ Note that the author has a background spanning a lot of the timespans/topics discussed - much work in multilingual NLP, translation, and more recently at DeepMind, Cohere, and Meta (in other words, someone with a great perspective on everything in the top article). Re: Machine Translation, note that Transformers were introdu…
Comment linking /nlp-imagenet/
on “When ChatGPT broke the field of NLP: An oral history”Submitted The secret ingredients of word2vec (2016) as a story · 191 points
HN discussion1. If you goal is to work in industry, don't do pure mathematics PhD. What works, and is doable in 20-30h/week is doing an applied PhD in an area related to your work. I know quite a few people who did it with great success. (Quite a few friends, and some accomplished people like https://ruder.io/. ) Also, as a person with PhD - spending 4-7 years in an irrelevant field is going to seriously hurt your industry caree…
Comment linking /
on “Reimagining the PhD”Submitted Word embeddings in 2017: Trends and future directions as a story · 158 points
HN discussionTangentially related if you're interested in keeping up to date with what's going on in the field: Sebastian Ruder's blog has many good posts on recent advancements in NLP (literature reviews, conference highlights): http://ruder.io/ The posts are concise and accessible enough that you can skim through them quickly. Then you can go check out the paper directly if something piques your interest.
Comment linking /
on “Natural Language Processing with Deep Learning”Submitted Do NLP Beyond English as a story · 148 points
HN discussion2017 version: https://www.ruder.io/word-embeddings-2017/
Comment linking /word-embeddings-2017/
on “The secret ingredients of word2vec (2016)”Submitted An overview of gradient descent optimization algorithms (2016) as a story · 135 points
HN discussionMachine learning researchers aren't working on language modeling because they want to enable fake news. They are working on it because it improves all downstream NLP tasks. See: http://ruder.io/nlp-imagenet/ . BERT, Elmo and XLNet all fall under this use case. For example if you're trying to recognize speech or translate some text, it helps a lot if you can start off producing something that is statistically grammat…
Comment linking /nlp-imagenet/
on “Salesforce releases language model bigger than GPT-2 large”Submitted The State of Transfer Learning in NLP as a story · 122 points
HN discussionNice, I was just starting to contemplate about how much work it would be to train a bert model from scratch. Context to understand the importance of this release: http://ruder.io/nlp-imagenet/ Though not named in the post, BERT is part of the same family of models as these.
Comment linking /nlp-imagenet/
on “TensorFlow Code for Google Research's BERT: Pre-Training Method for NLP Tasks”Submitted Optimization for Deep Learning Highlights as a story · 102 points
HN discussionFor those who don't know, this article is by Sebastian Ruder, who has a great blog [1]. His posts are often pretty deep dives of a particular area of interest within Natural Language Processing. A favorite of mine from last year gave a summary of the state-of-the-art in "Word Embeddings" [2], which I would recommend for anyone interested in the field. [1] http://ruder.io/#open [2] http://ruder.io/word-embeddings-201…
Comment linking /word-embeddings-2017/
on “A Review of the Neural History of Natural Language Processing”Submitted An overview of gradient descent optimization algorithms as a story · 62 points
HN discussionHere[1] is an article describing the same, written by the author himself. [1]: http://ruder.io/optimizing-gradient-descent/index.html
Comment linking /optimizing-gradient-descent/index.html
on “An overview of gradient descent optimization algorithms”Submitted An overview of gradient descent optimization algorithms as a story · 43 points
HN discussionA good run-down of the different algorithms: https://ruder.io/optimizing-gradient-descent/ I think people favour adaptive learning rate options like Adam in practice since they generally do seem to perform well, and are often less sensitive to initial conditions and the exact hyper-parameters used. There will always be people who like to test N optimizers with parameter sweeps to squeeze a tiny bit of extra performa…
Comment linking /optimizing-gradient-descent/
on “What Is Machine Learning Anyway?”Submitted Catalog of Gradient Descent Optimization Algorithms as a story · 6 points
HN discussionhttp://ruder.io/optimizing-gradient-descent/
Comment linking /optimizing-gradient-descent/
on “Show HN: AdaBound, an optimizer that trains as fast as Adam and as good as SGD”Submitted A survey of cross-lingual word embedding models (2016) as a story · 4 points
HN discussionWhich one? Temporally Recurrent Optimal Learning [0] [1] [2] is the only one i know about, but it wasn't that famous afaik. [0] https://www.ruder.io/highlights-nips-2016/ [1] https://medium.com/the-mission/rocket-ai-2016s-most-notoriou... [2] https://securelist.com/rocket-ai-and-the-next-generation-of-...
Comment linking /highlights-nips-2016/
on “Cryptographers Solve Decades-Old Privacy Problem”Submitted The Evolving Landscape of LLM Evaluation as a story · 3 points
HN discussion"similar" means similar topic or similar interactivity? similar interactivity: https://epiphany.pub/@shi-yan/introduction https://www.smart-biology.com/ https://mathigon.org/ http://immersivemath.com/ila/index.html https://textbooks.math.gatech.edu/ila/ similar topic https://ruder.io/
Comment linking /
on “Ask HN: Do you know any other website similar to distill.pub?”Submitted Recent Advances in Language Model Fine-Tuning (2021) as a story · 3 points
HN discussionThere's definitely active research on it. But here's the basic recipe that gets state-of-the-art accuracy on most tasks. It's been around for 5 years or so now, which is why I said "just". You take an encoder transformer architecture like BERT and train it on a language modelling objective. Then you discard the part of the network that does the token prediction, and stick some randomly initialised network that does…
Comment linking /nlp-imagenet/
on “Against LLM Maximalism”Submitted NLP's ImageNet moment has arrived (2018) as a story · 3 points
HN discussionSubmitted 10 Exciting Ideas of 2018 in NLP as a story · 3 points
HN discussionSubmitted A collection of interesting research directions around transfer learning and NLP as a story · 3 points
HN discussion
