Top all-time · #1470
Variance Explained
Personal blog · varianceexplained.orgdataprogramming
Variance Explained is a personal blog at varianceexplained.org about data and programming. Since 2015, 47 different Hacker News readers have linked it 86 times, most often “Text analysis of Trump's tweets shows he writes only the (angrier) Android half”. The last post was in 2021, but the archive is still widely read.
Too few sampled comments (23) to say how often HN recommends it.?
Why HN links it
HN readers link David Robinson's varianceexplained.org for posts on data science, A/B testing, R and empirical Bayes, including the Trump tweets analysis.
Summarised by Claude from a sample of 23 HN comments that link here. Each comment is marked recommending, critical or neutral (citing without judging); how this works.
Blog on R, statistics, and education
Most-linked posts?
- Text analysis of Trump's tweets shows he writes only the (angrier) Android half10 people · 10 mentions · first linked 2016-08-09
- Is Bayesian A/B Testing Immune to Peeking? Not5 people · 6 mentions · first linked 2017-01-20
- Scientific debt4 people · 4 mentions · first linked 2018-05-10
- What's the difference between data science, machine learning, and AI?4 people · 4 mentions · first linked 2018-01-09
- Who Wrote the Anti-Trump New York Times Op-Ed? Using Tidytext4 people · 4 mentions · first linked 2018-09-06
- Teach the tidyverse to beginners in R3 people · 4 mentions · first linked 2017-07-05
- How we built Tagger News: machine learning on a tight schedule2 people · 3 mentions · first linked 2017-05-15
- Introduction to Empirical Bayes (book)2 people · 2 mentions · first linked 2017-02-08
- Machine Learning in a Hurry2 people · 2 mentions · first linked 2021-07-22
- Does sentiment analysis work? A tidy analysis of Yelp reviews2 people · 2 mentions · first linked 2016-07-23
What Hacker News says?
I found this a nice way to communicate the difference between AI, ML and Data Science [0]. TL;DR AI produces actions ML produces predictions Data Science produces insights [0] http://varianceexplained.org/r/ds-ml-ai/
Comment linking /r/ds-ml-ai/
on “Apple Hires Google’s A.I. Chief”Submitted One Year as a Data Scientist at Stack Overflow as a story · 189 points
HN discussionSometimes a good analogy can take the place of a visualization [1], albeit I did imagine a distribution getting more concentrated. [1] http://varianceexplained.org/statistics/beta_distribution_an...
Comment linking /statistics/beta_distribution_and_baseball/
on “Seeing Theory: A visual introduction to probability and statistics (2017)”Submitted A Million Lines of Bad Code as a story · 175 points
HN discussionThat would go against most public accounts of how Trump has been using his phone for tweeting. For one thing, Obama's personal account (i.e. not @POTUS, but @BarackObama) explicitly says in its profile: "This account is run by Organizing for Action staff. Tweets from the President are signed -bo. [0] No such disclaimer has been made about Trump's account, which tweets with the same characteristics and tone as it did…
Comment linking /r/trump-tweets/
on “Twitter’s Chief Technology Officer to Leave Company”Submitted Gender and verbs across 100K stories: a tidy analysis as a story · 159 points
HN discussionI created an open source A/B test framework[0], which also uses Bayesian analysis on the dashboard. IANAS(tatistician), but from what I understand it’s still better to plan the check point in advance, rather than stop when reaching significance. A couple of articles worth reading [1] [2] (can’t exactly vouch for their validity but seem to make some good arguments that appear thought out) [0] https://github.com/Aleph…
Comment linking /r/bayesian-ab-testing/
on “p-Hacking and False Discovery in A/B Testing”Submitted Words growing or shrinking in Hacker News titles: a tidy analysis as a story · 118 points
HN discussionI was thinking of the same article, and found it, I think [1]? There was a higher-rated link also posted around the same time with the same post content [2]. I also found a follow up post [3] while looking for [1]. [1] http://varianceexplained.org/r/trump-tweets/ [2] https://www.r-bloggers.com/text-analysis-of-trumps-tweets-co... [3] http://varianceexplained.org/r/trump-followup/
Comment linking /r/trump-followup/
on “Devices Which Track Cellphones, Intercept Calls Found All Over DC, MD, VA”Submitted Who Wrote the Anti-Trump New York Times Op-Ed? Using Tidytext as a story · 114 points
HN discussionWe ran into this problem often teaching R when I was in grad school. However, in the years since, the tidyverse has put strong focus on getting users manipulating data even if they don't know how to define a function, etc..! Here's a useful post, comparing the classic approach you mention to an alternative http://varianceexplained.org/r/teach-tidyverse/
Comment linking /r/teach-tidyverse/
on “What's Next for R?”Submitted Data Analysis and Visualization Using R (2014) as a story · 95 points
HN discussion> For that, I might look at another source of data, Stack Overflow Careers profiles, and see which technologies tend to be used by the same developers > http://varianceexplained.org/images/network2.jpeg This shows "git" and "github" in a separate cluster from "C++" and "Python"? I don't understand this. These tools are used regardless of what other technologies are being used. For example, there are many Python and…
Comment linking /images/network2.jpeg
on “One Year as a Data Scientist at Stack Overflow”Submitted Analyzing networks of characters in 'Love Actually' as a story · 83 points
HN discussionJust to expand on this, David Robinson quantified this a while back: http://varianceexplained.org/r/bayesian-ab-testing/ He ran some simulated A/B tests using my Bayesian A/B testing technique (which is now powering VWO). He showed that while peeking does ensure that loss < specified threshold, if you don't peek the loss will be even lower. So although peeking is still valid, that validity does come at a cost.
Comment linking /r/bayesian-ab-testing/
on “Most Winning A/B Test Results Are Illusory [pdf]”Submitted What are the most polarizing programming languages? as a story · 78 points
HN discussionDirect link to a scatter plot (x axis: quantity, y axis: gender): http://varianceexplained.org/figs/2017-04-27-tidytext-gender...
Comment linking /figs/2017-04-27-tidytext-gender-plots/total_log_ratio_scatter-1.png
on “Gender and verbs across 100K stories: a tidy analysis”Submitted Text analysis of Trump's tweets shows he writes only the (angrier) Android half as a story · 74 points
HN discussionThank you for explanation. If I understand correctly, you use this approach to increase sensitivity (compared to NHST) using the same data . Most probably, in your case, higher sensitivity (or power) comes at the cost of higher type I error rate. And this might be fine. Sometimes making more changes and faster is more important than false positives. In this case, you can just use a higher p-value threshold in the NH…
Comment linking /r/bayesian-ab-testing/
on “Show HN: Tea-tasting, a Python package for the statistical analysis of A/B tests”Submitted How we built Tagger News: machine learning on a tight schedule as a story · 71 points
HN discussionHe probably does use his phone to tweet http://varianceexplained.org/r/trump-tweets/
Comment linking /r/trump-tweets/
on “On Wall Street, a high-ranking few still avoid email”Submitted Why I use ggplot2 as a story · 64 points
HN discussionat the end of the comments of the post http://varianceexplained.org/r/teach-tidyverse/ compare base R: var1 <- "pounds" var2 <- "wt" mtcars[[var1]] <- mtcars[[var2]]/1000 Now hadley solution comment: You can now express that sort of thing fairly elegantly with dplyr: var1 <- sym("pounds") var2 <- sym("wt") mtcars <- mutate(mtcars,!!var1 := !!var/100) Watch your steps: !!, :=, sym. Thats a lot of added complexity to…
Comment linking /r/teach-tidyverse/
on “An opinionated view of the Tidyverse “dialect” of the R language”Submitted The 'Spelling Bee Honeycomb' puzzle: efficient computation in R as a story · 48 points
HN discussionA related post is http://varianceexplained.org/r/teach-tidyverse/
Comment linking /r/teach-tidyverse/
on “An opinionated view of the Tidyverse “dialect” of the R language”Submitted The 'lost boarding pass' puzzle: efficient simulation in R as a story · 35 points
HN discussionCan we link to the original blog post please? http://varianceexplained.org/r/trump-tweets/ Not only is it nicer to the author, it's far better formatted as well.
Comment linking /r/trump-tweets/
on “R: text analysis of Trump’s tweets”Submitted Text analysis of Trump's tweets: he writes only the (angrier) Android half as a story · 16 points
HN discussionAlthough tangentially linked to in the article, David Robinson's Introduction to Empirical Bayes[1] is also an excellent resource. It deals primarily with beta-binomial distributions. [1] http://varianceexplained.org/r/empirical-bayes-book/
Comment linking /r/empirical-bayes-book/
on “Empirical Bayes for multiple sample sizes”Submitted Exploring handwritten digit classification: a tidy analysis of the MNIST dataset as a story · 15 points
HN discussion
