Top all-time · #591
Wes McKinney
Personal blog · wesmckinney.com · on HN as wesmdataprogramming
Wes McKinney's personal blog at wesmckinney.com covers data and programming. Since 2011, 103 different Hacker News readers have linked it 139 times, most often “Apache Arrow and the 10 Things I Hate About Pandas (2017)”. No feed was found, so read it on the site itself.
Sometimes recommended · 7 of 39 sampled comments recommend it?
Why HN links it
HN readers cite Wes McKinney's posts on pandas' limits (the 'pandas internals' post), his whirlwind pandas tour and his book, and some link his Arrow and Polars-era writing as context.
- Recommended for · 7 comments
- 'Pretty good example' of pandas use; 'definitely the standard' book; a 'helpful' book; 'very successful outcome' for a newcomer.
Summarised by Claude from a sample of 39 HN comments that link here. Each comment is marked recommending, critical or neutral (citing without judging); how this works.
Most-linked posts?
- Apache Arrow and the 10 Things I Hate About Pandas (2017)16 people · 16 mentions · first linked 2020-07-27
- The Mythical Agent-Month9 people · 11 mentions · first linked 2026-02-17
- From Human Ergonomics to Agent Ergonomics8 people · 9 mentions · first linked 2026-01-20
- Wes McKinney: The Road to Composable Data Systems7 people · 7 mentions · first linked 2023-09-02
- 2017 Outlook: Pandas, Arrow, Feather, Parquet, Spark, Ibis6 people · 6 mentions · first linked 2016-12-27
- Python for Data Analysis, 3rd Edition5 people · 7 mentions · first linked 2022-07-02
- Apache Arrow and the “Things I Hate About Pandas”5 people · 5 mentions · first linked 2017-09-21
- Why pandas users should be excited about Apache Arrow4 people · 5 mentions · first linked 2016-02-25
- Why I’m not on the Julia bandwagon (yet) (2012)3 people · 3 mentions · first linked 2012-05-03
- Msgvault: Fast Gmail search from the author of Pandas3 people · 3 mentions · first linked 2026-02-02
What Hacker News says?
Your first paragraph makes perfect sense! I was nodding along. But then your concluding sentence was a bit of a record scratch for me. This all worked as intended! You knew what the project was about - "data frames" - and what might make it attractive to you - the performance claims - and then you went and followed exactly the right path to get the context you needed to understand what's going on with it. It's a big…
Comment linking /book/
on “Polars”Submitted Apache Arrow and the “Things I Hate About Pandas” as a story · 394 points
HN discussionI really think this is a misunderstanding of the purpose of different kinds of documentation. The documentation of a new tool for a mature technique is just not the primary place to focus on writing a beginners' tutorial / course on using that technique. Certainly, "the more the merrier" is a good mantra for documentation, so if they do add such material, all the better. But it is very sensible for it to not be the…
Comment linking /book/
on “Polars”Submitted Python for Data Analysis, 3rd Edition as a story · 271 points
HN discussionYes, it's annoying negative feature of many tech products. Of course it's natural to want to speak to your target audience (in this case, data scientists who like Pandas but find it annoyingly slow/inflexible), but it's quite alienating to newbies who might otherwise become your most enthusiastic customers. I am the target audience for Polars and have been meaning to try it for several months, but I keep procrastina…
Comment linking /book/
on “Polars”Submitted Leaving NYC for Nashville as a story · 209 points
HN discussionWes McKinney posted a 10-minute "whirlwind tour of pandas" [0] several years ago that still serves as a pretty good example of what is possible for a practiced pandas user. [0] http://wesmckinney.com/blog/whirlwind-tour-of-pandas-in-10-m...
Comment linking /blog/whirlwind-tour-of-pandas-in-10-minutes/
on “JupyterLab is ready for users”Submitted Ursa Labs: an innovation lab for open source data science as a story · 170 points
HN discussionFor use as a file format, where one priority is to compress columnar data as well as possible, the practical difference between Arrow (via Feather?), Parquet, and ORC is still somewhat vague to me. During my last investigation, I got the impression that Arrow worked great as a standard, interoperable, in-memory columnar format, but didn't compress nearly as well as ORC or Parquet due to lack of RLE and other compres…
Comment linking /blog/arrow-columnar-abadi
on “Apache Arrow 3.0”Submitted Linux Developer Laptops: Dell's Precision 5500 series reigns supreme (2019) as a story · 129 points
HN discussionI could be misremembering but I seem to recall Wes McKinney saying in some talk that rather than rewrite Pandas to be Arrow-backed it will probably eventually be replaced by newer Arrow-backed libraries some of which might have pandas-like apis. I think the idea was that pandas API is too large and the library too widely used for it to be practical to correct some of the design problems people have mentioned. He'd s…
Comment linking /blog/apache-arrow-pandas-internals/
on “I wrote one of the fastest DataFrame libraries”Submitted 2017 Outlook: Pandas, Arrow, Feather, Parquet, Spark, Ibis as a story · 78 points
HN discussionCheck out the post from author of pandas on pandas, pandas2 and arrow [1]. This might answer your question on what is the significance of arrow, given today pandas are kind of basic ingredient in scientific computing, AI and ML. [1] https://wesmckinney.com/blog/apache-arrow-pandas-internals/
Comment linking /blog/apache-arrow-pandas-internals/
on “Apache Arrow 1.0”Submitted High Performance Database Joins in Python with pandas as a story · 75 points
HN discussionHere is an example: https://wesmckinney.com/blog/avoid-unsigned-integers/
Comment linking /blog/avoid-unsigned-integers/
on “Integers in C quiz”Submitted A Roadmap for Rich Scientific Data Structures in Python as a story · 70 points
HN discussionThe first edition of this book was a game changer. I still have my hardcopy paperback but haven’t turned to it in years. It’s probably largely out of date but I look at it fondly. Glad to see the material updated with new APIs and Python 3.10. The open access is very nice because it can easily be used for teaching and reference. I would like to see some of Wes’ cautionary tales included into the book. Pandas feels s…
Comment linking /blog/apache-arrow-pandas-internals/
on “Python for Data Analysis, 3rd Edition”Submitted Why I’m not on the Julia bandwagon (yet) (2012) as a story · 36 points
HN discussion10x reflects both experience and expert recommendations. You may recognize the author [1]: > Nowadays, my rule of thumb for pandas is that you should have 5 to 10 times as much RAM as the size of your dataset [1] https://wesmckinney.com/blog/apache-arrow-pandas-internals/
Comment linking /blog/apache-arrow-pandas-internals/
on “Pyston v2: Faster Python”Submitted Vbench: run Python benchmarks throughout your repository history as a story · 33 points
HN discussionWes's book is definitely the standard. But I would hold off buying one right now - he's currently working on a (much-needed) second edition, coming out next year ( http://wesmckinney.com/ ). Joel's book is a great resource for preparing for interviews or learning really basic stuff and less of an introduction to the tools.
Comment linking /
on “R for Data Science”Submitted Pivot tables in Python with pandas as a story · 15 points
HN discussionRelated to this, below someone posted a link to the blog post of Wes McKinney where he discussed Pandas limitations and how PyArrow works around these > 4. Doing missing data right > All missing data in Arrow is represented as a packed bit array, separate from the rest of the data. This makes missing data handling simple and consistent across all data types. You can also do analytics on the null bits (AND-ing bitmap…
Comment linking /blog/apache-arrow-pandas-internals/
on “Farewell Pandas, and thanks for all the fish”Submitted Apache Arrow and the “Things I Hate About Pandas” as a story · 13 points
HN discussionAbout time. It always surprises me how long pandas has been able to hold on. Wes McKinney talked about pandas' limitations back in 2017 https://wesmckinney.com/blog/apache-arrow-pandas-internals/
Comment linking /blog/apache-arrow-pandas-internals/
on “Farewell Pandas, and thanks for all the fish”Submitted Joining Forces for an Arrow-Native Future as a story · 12 points
HN discussionsome sources: - https://voltrondata.com/codex/a-new-frontier (links out to others) - https://wesmckinney.com/blog/looking-back-15-years/ in short you can think of a DB as at least 3 decoupled subsystems: UI, compute (query engine), storage. DuckDB has a query engine and storage format, and several UIs (SQL, Python, etc.). Trino is only a query engine (and UIs, everything has UIs). Polars has a query engine. DataFusi…
Comment linking /blog/looking-back-15-years/
on “The two versions of Parquet”Submitted Why I'm not on the Julia bandwagon (yet) as a story · 7 points
HN discussionWes McKinney (author of Pandas, but also co-author of a host of other major data formats/tools like Ibis, Arrow, Parquet, Feather), wrote "10 Things I Hate About pandas" https://wesmckinney.com/blog/apache-arrow-pandas-internals/ It was one of his earlier projects, and it "stuck". It was one of the first popular dataframe libraries for Python and it filled that niche for years (alongside dplyr/data.table in R), espe…
Comment linking /blog/apache-arrow-pandas-internals/
on “DuckDB 1.1.0 Released”Submitted The Problem with the Data Science Language Wars as a story · 6 points
HN discussion
