Top all-time · #6511
Andy Grove
Personal blog · andygrove.iodataprogrammingsystems
Andy Grove's personal blog at andygrove.io covers data, programming and systems. Since 2018, 12 different Hacker News readers have linked it 22 times, most often “Why does musl make my Rust code so slow?”. The last post was in 2023, but the archive is still widely read.
Too few sampled comments (7) to say how often HN recommends it.?
Why HN links it
HN readers link Andy Grove's posts on building DataFusion, Ballista and other Rust Arrow-based query engine projects, and his guide to how query engines work.
Summarised by Claude from a sample of 7 HN comments that link here. Each comment is marked recommending, critical or neutral (citing without judging); how this works.
Software engineer specializing in query engines and distributed systems
Most-linked posts?
- Why does musl make my Rust code so slow?4 people · 4 mentions · first linked 2020-05-05
- Ballista: Distributed Compute with Rust, Apache Arrow, and Kubernetes2 people · 2 mentions · first linked 2019-07-17
- First Benchmark for DataFusion (Rust) vs. Apache Spark for a Trivial Workload2 people · 2 mentions · first linked 2018-03-16
- /2019/04/datafusion-0.13.0-benchmarks/2 people · 2 mentions · first linked 2019-07-17
What Hacker News says?
I watched the video and it seems they are statically linking atop musl to build their lightweight VM layer. I guess the container app itself might use glibc, but will the musl build for the VM itself cause downstream performance issues? I'm no expert in virtualization to be able to understand if this should be a concern or not. See also: - https://edu.chainguard.dev/chainguard/chainguard-images/abou... - https://and…
Comment linking /2020/05/why-musl-extremely-slow/
on “Apple announces Foundation Models and Containerization frameworks, etc”Submitted Ballista: Distributed Compute with Rust, Apache Arrow, and Kubernetes as a story · 194 points
HN discussionMy naive interpretation - The canonical Apache Arrow implementation is written in C++ with multiple language bindings like PyArrow. The Rust bindings for Apache Arrow re-implemented the Arrow specification so it can be used as a pure Rust implementation. Andy Grove [1] built two projects on top of Rust-Arrow: 1. DataFusion, a query engine for Arrow that can optimize SQL-like JOIN and GROUP BY queries, and 2. Ballist…
Comment linking /projects/
on “I wrote one of the fastest DataFrame libraries”Submitted Rust DataBase Connectivity (RDBC) as a story · 167 points
HN discussion> datafusion > blazingly fast I’m going to need to see a citation for that. Last I checked, it was being beaten by Apache Spark in non-memory constrained scenarios [0]. This may be “blazingly fast” compared to Pandas or something, but it’s still leaving a TON of room on the table performance-wise. There’s a reason why Databricks found it necessary to redirect their Spark backend to a custom native query engine [1].…
Comment linking /2019/04/datafusion-0.13.0-benchmarks/
on “Run SQL on CSV, Parquet, JSON, Arrow, Unix Pipes and Google Sheet”Submitted Why does musl make my Rust code so slow? as a story · 154 points
HN discussionThere is also a more recent blog post which perhaps led to the project being posted here (I am guessing). https://andygrove.io/2021/01/ballista-2021/
Comment linking /2021/01/ballista-2021/
on “Ballista: Distributed compute platform implemented in Rust using Apache Arrow”Submitted DataFusion (Rust) vs. Apache Spark – First Benchmarks as a story · 5 points
HN discussionBlog post that started the project. Worth reading. https://andygrove.io/2019/07/announcing-ballista/
Comment linking /2019/07/announcing-ballista/
on “Ballista: Distributed compute platform implemented in Rust using Apache Arrow”Submitted First Benchmark for DataFusion (Rust) vs. Apache Spark for a Trivial Workload as a story · 5 points
HN discussionSome good points. Some incredible engineering has gone into Spark to work around the fact that it runs on the JVM. Memory overhead of Spark particularly (not just JVM) is very high. In some cases close to 100x more memory than equivalent query execution with DataFusion [0]. Also you might be interested to see my original blog post with some of my thoughts on this [1]. [0] https://andygrove.io/2019/04/datafusion-0.13…
Comment linking /2018/01/rust-is-for-big-data/
on “Ballista: Distributed Compute with Rust, Apache Arrow, and Kubernetes”Submitted First Benchmark of DataFusion (Rust) vs. Apache Spark for a Trivial Workload as a story · 3 points
HN discussionThis is a personal open source project. I'm not sure I'm "marketing" it since I make zero dollars from this work. For benchmarks, check out the past 18 months of posts on my blog. Here is the most recent: https://andygrove.io/2019/04/datafusion-0.13.0-benchmarks/
Comment linking /2019/04/datafusion-0.13.0-benchmarks/
on “Ballista: Distributed Compute with Rust, Apache Arrow, and Kubernetes”Submitted Why does musl make my Rust code so slow? (2020) as a story · 2 points
HN discussionMy goal is to kick start some things around the Rust language for data science / data analytics and projects like this help to showcase what Rust can do. See my original blog post for more context: https://andygrove.io/2018/01/rust-is-for-big-data/
Comment linking /2018/01/rust-is-for-big-data/
on “Ballista: Distributed Compute with Rust, Apache Arrow, and Kubernetes”Submitted Looking for help with multi-threaded Rust code (Parallel query execution) as a story · 2 points
HN discussionHe is not the first, Andy Grove worked on a similar thing last years, but he worked mostly alone, and without community you cannot take off such ambitious project: https://andygrove.io/2018/11/datafusion-2019/
Comment linking /2018/11/datafusion-2019/
on “The Road to Scala 3”Submitted Ballista Distributed Compute: One Year Later as a story · 1 points
HN discussionI'd also recommend "How Query Engines Work" for a good practical exercise in the topic: https://andygrove.io/how-query-engines-work/ . I think one should go already have a high-level overview of what individual parts make up a query engine and what there purpose is, as it only touches on that lightly.
Comment linking /how-query-engines-work/
on “Joins 13 Ways”Submitted Rust Big Data Benchmarks as a story · 1 points
HN discussionMusl is much less optimized than glibc, random example: https://andygrove.io/2020/05/why-musl-extremely-slow/
Comment linking /2020/05/why-musl-extremely-slow/
on “We reduced a real Node.js production Docker image from 1.2GB to 78MB”
