Top all-time · #978
Jack Vanlightly
Personal blog · jack-vanlightly.comsystemsdataprogramming
Jack Vanlightly's personal blog at jack-vanlightly.com covers systems, data and programming. Since 2018, 71 different Hacker News readers have linked it 113 times, most often “On the Future of Cloud Services and BYOC”. It is still writing: the last post was on 2026-08-28, and you can follow it by feed.
{ dist sys, formal verification, event streaming }
Most-linked posts?
- On the Future of Cloud Services and BYOC8 people · 8 mentions · first linked 2023-09-26
- Why Apache Kafka Doesn't Need Fsync to Be Safe7 people · 7 mentions · first linked 2023-04-24
- Understanding How Apache Pulsar Works5 people · 5 mentions · first linked 2018-10-17
- Understanding Delta Lake's consistency model3 people · 4 mentions · first linked 2024-04-29
- A mental model of the standard ClickHouse architecture3 people · 3 mentions · first linked 2024-01-23
- Building a “Simple” Distributed System – Formal Verification3 people · 3 mentions · first linked 2019-01-29
- The Advantages of Queues on Logs3 people · 3 mentions · first linked 2023-10-04
- Demystifying Determinism in Durable Execution2 people · 2 mentions · first linked 2025-11-24
- Is sequential IO dead in the era of the NVMe drive?2 people · 2 mentions · first linked 2023-05-09
- Understanding Apache Fluss2 people · 2 mentions · first linked 2025-09-02
What Hacker News says?
I've been dreaming of Apple Notes and Obsidian doing what the author suggests. The approach seems similar to Delta Lake's consistency model, which is using object storage like S3, and yet allows concurrent writers and readers: https://jack-vanlightly.com/analyses/2024/4/29/understanding... .
Comment linking /analyses/2024/4/29/understanding-delta-lakes-consistency-model
on “Local First, Forever”Submitted Write for others but mostly for yourself (2022) as a story · 184 points
HN discussionThis is one of the best overviews on Pulsar with comparisons to Kafka: https://jack-vanlightly.com/blog/2018/10/2/understanding-how...
Comment linking /blog/2018/10/2/understanding-how-apache-pulsar-works
on “Apache Pulsar is an open-source distributed pub-sub messaging system”Submitted Building a “Simple” Distributed System – Formal Verification as a story · 151 points
HN discussionIt's a persistent store, so that would be different from Redis Pub-Sub. Compared to RabbitMQ, Pulsar seems to favor strong ordering and protection from message loss. This blog post offers some more info and leaders to other posts comparing Pulsar to RabbitMQ and Kafka: https://jack-vanlightly.com/blog/2018/10/2/understanding-how...
Comment linking /blog/2018/10/2/understanding-how-apache-pulsar-works
on “Apache Pulsar is an open-source distributed pub-sub messaging system”Submitted Beyond indexes: How open table formats optimize query performance as a story · 95 points
HN discussionI highly recommend all of Jack's blog posts about RabbitMQ - https://jack-vanlightly.com/blog/tag/RabbitMQ Jack works with me on the RabbitMQ core engineering team. We've been hard at work to address a lot of the issues brought up in comments here. It's worth it to try out our latest releases. The engineering team is very active with the community and takes all constructive, helpful (i.e. reproducible) feedback seri…
Comment linking /blog/tag/RabbitMQ
on “An introduction to RabbitMQ”Submitted Understanding How Apache Pulsar Works as a story · 69 points
HN discussionThanks for all your work on RabbitMQ, and for your great blog posts about it and other messaging systems. For anyone who wants to understand the potential complexities of HA RabbitMQ, spend some time reading https://jack-vanlightly.com/blog/2018/8/31/rabbitmq-vs-kafka...
Comment linking /blog/2018/8/31/rabbitmq-vs-kafka-part-5-fault-tolerance-and-high-availability-with-rabbitmq
on “An introduction to RabbitMQ”Submitted A Fork in the Road: Deciding Kafka's Diskless Future as a story · 59 points
HN discussionWe fixed that particular issue: https://jack-vanlightly.com/blog/2023/8/17/kafka-kip-966-fix...
Comment linking /blog/2023/8/17/kafka-kip-966-fixing-the-last-replica-standing-issue
on “Kafka is Fast – I'll use Postgres”Submitted Dismantling ELT: The Case for Graphs, Not Silos as a story · 35 points
HN discussionNo. Redpanda has made a lot of noise about this over the years [0], and Confluent's Jack Vanlightly has responded in a fair bit of detail [1]. [0]: https://www.redpanda.com/blog/why-fsync-is-needed-for-data-s... [1]: https://jack-vanlightly.com/blog/2023/4/24/why-apache-kafka-...
Comment linking /blog/2023/4/24/why-apache-kafka-doesnt-need-fsync-to-be-safe
on “Jepsen: NATS 2.12.1”Submitted A Conceptual Model for Storage Unification as a story · 27 points
HN discussionThey keep using the core Postgre while they touch the storage layer to works with S3. Can try ro read more here https://jack-vanlightly.com/analyses/2023/11/15/neon-serverl...
Comment linking /analyses/2023/11/15/neon-serverless-postgresql-asds-chapter-3
on “Databricks in talks to acquire startup Neon for about $1B”Submitted The Sisyphean struggle and the new era of data infrastructure as a story · 17 points
HN discussionThanks for feedback. The link is broken for me, but I think you are referring to this page? https://jack-vanlightly.com/analyses/2024/4/29/understanding... Indeed, the approach is similar. Especially the separation of assets (they call it "Data files") from the log ("Data log") is something I consider being a good choice.
Comment linking /analyses/2024/4/29/understanding-delta-lakes-consistency-model
on “Resilient Sync for Local First”Submitted Building a “Simple” Distributed System – The What as a story · 9 points
HN discussionThe approach seems similar to Delta Lake's consistency model, using object storage like S3, and yet allows concurrent writers and readers: https://jack-vanlightly.com/analyses/2024/4/29/understanding...
Comment linking /analyses/2024/4/29/understanding-delta-lakes-consistency-model
on “Resilient Sync for Local First”Submitted Learning and Reviewing System Internals: Tactics and Psychology as a story · 5 points
HN discussionhttps://jack-vanlightly.com/blog/2018/10/2/understanding-how... goes into this. > The data of a given topic is spread across multiple Bookies. The topic has been split into Ledgers and the Ledgers into Fragments and with striping, into calculatable subsets of fragment ensembles. When you need to grow your cluster, just add more Bookies and they’ll start getting written to when new fragments are created. No more Kafk…
Comment linking /blog/2018/10/2/understanding-how-apache-pulsar-works
on “Pulsar – an open-source distributed pub-sub messaging platform”Submitted Kafka vs. Redpanda Performance – Do the claims add up? as a story · 5 points
HN discussionThis is a great review of Pulsar by Jack Vanlightly. If you haven't read it, I highly recommend it: https://jack-vanlightly.com/blog/2018/10/2/understanding-how... > The thing I don't like is that we also need BookKeeper, but yeah. Word on the street is now you can swap the consensus module in ZooKeeper. [update: just re-checked Pulsar's dev channel. Apparently one can use etcd instead of zookeeper.] > sirixdb Sirix…
Comment linking /blog/2018/10/2/understanding-how-apache-pulsar-works
on “Building a Relational Database Using Kafka”Submitted Have Your Iceberg Cubed, Not Sorted: Meet Qbeast, the OTree Spatial Index as a story · 4 points
HN discussionFor more details on pulsar and comparisons to similar systems see this series of blog posts from a rabbitmq developer. https://jack-vanlightly.com/blog/2018/10/2/understanding-how... https://jack-vanlightly.com/blog/2018/10/21/how-to-not-lose-... https://jack-vanlightly.com/blog/2018/10/25/testing-producer... https://jack-vanlightly.com/blog/2019/9/4/a-look-at-multi-to...
Comment linking /blog/2019/9/4/a-look-at-multi-topic-subscriptions-with-apache-pulsar
on “Pulsar vs. Kafka”Submitted Why I'm not a fan of zero-copy Apache Kafka-Apache Iceberg as a story · 4 points
HN discussionAWS should fork Pulsar and put out a v2 streaming product. Kinesis is kind of crappy (IMO) and doesn't seem to be improving much. If you look at the Pulsar architecture and feature set you can tell that it was designed very much with this in mind (something that large scale cloud providers can integrate with their infinitely scalable storage and compute systems). It's not all hype either, according to this post http…
Comment linking /blog/2018/10/21/how-to-not-lose-messages-on-an-apache-pulsar-cluster
on “Pulsar vs. Kafka”Submitted Understanding Apache Fluss as a story · 4 points
HN discussionThanks for the report - we just deployed fix for the same. No lock-in means it’s 100% open source (PromQL) compatible. You can swap out vendors or move to self-hosted open source solutions should you need to move away from Oodle. When you migrate out, you get to export all your data, dashboards and alerts. you don't need to make any code changes. We support bringing your own bucket (BYOB) for large enterprise custom…
Comment linking /blog/2023/9/25/on-the-future-of-cloud-services-and-byoc
on “Show HN: Oodle – serverless, fully-managed, drop-in replacement for Prometheus”Submitted A mental model of the standard ClickHouse architecture as a story · 4 points
HN discussion
