Top all-time · #8000
Robin Linacre
Personal blog · robinlinacre.com · on HN as RobinLdataprogramming
Robin Linacre's personal blog at robinlinacre.com covers data and programming. Since 2020, 10 different Hacker News readers have linked it 81 times, most often “Building Accurate Address Matching Systems”. It is still writing: the last post was on 2026-10-19, and you can follow it by feed.
Too few sampled comments (1) to say how often HN recommends it.?
Why HN links it
The linked posts cover Amazon S3 Tables and lakehouse use with DuckDB, and recommending DuckDB. The only comment is a bare reference to an episode and transcript cited in the blog.
Summarised by Claude from a sample of 1 HN comments that link here. Each comment is marked recommending, critical or neutral (citing without judging); how this works.
Probabilistic record linkage, Data Deduplication, Data Science, Engineering and the Environment
Most-linked posts?
- Building Accurate Address Matching Systems3 people · 5 mentions · first linked 2025-06-30
- Why parquet files are my preferred API for bulk open data2 people · 8 mentions · first linked 2023-01-09
- Why DuckDB is my first choice for data processing2 people · 5 mentions · first linked 2025-03-16
- Demystifying Apache Arrow2 people · 5 mentions · first linked 2021-02-01
- Super-fast deduplication of large datasets using Splink and DuckDB2 people · 4 mentions · first linked 2024-01-19
What Hacker News says?
There's also two other important limitations to using an LLM and just providing it with pairs of records. It does not know enough about the records in the context of the overall dataset: - what is the data quality and to what extent do we expect a errors in some fields - how unusual are certain values such as names in the context of the dataset as a whole, e.g. some names would be very common in some countries but r…
Comment linking /fellegi_sunter_accuracy/
on “Introducing System One Models and Jev”Submitted Building Accurate Address Matching Systems as a story · 13 points
HN discussionObservable is such an incredible, powerful and enjoyable tool. I use it heavily, including to power my blog. I love it, but I've always had a slight concern about needing to rely on the Observable notebook website. So this has really made my day. To give a sense of the kind of performant, statically hosted interactive content that has only really been within my reach since using Observable, here are some examples: H…
Comment linking /partial_match_weights/
on “Observable 2.0, a static site generator for data apps”Submitted Super-fast deduplication of large datasets using Splink and DuckDB as a story · 5 points
HN discussionI wrote a bit about this from the perspective of a data scientist here: https://www.robinlinacre.com/demystifying_arrow/ I cover some of the use cases, but more importantly try and explain how it all fits together, justifying why - as another commenters has said - it's the most important thing happening in the data ecosystem right now. I wrote it because i'd heard a lot about Arrow, and even used it quite a lot, but…
Comment linking /demystifying_arrow/
on “Apache Arrow 3.0”Submitted Building accurate postal address matching systems as a story · 4 points
HN discussionAppreciate the kind words. There are a variety of people still use Spark, but this is usually due to constraints on their platform/infrastructure, not because it performs better. For instance people on Databricks can sometimes find it hard to use DuckDB on large machines, or they're just more familiar with Spark. For a new user who has flexibility over where to run their workloads, we'd always recommend DuckDB. This…
Comment linking /recommend_duckdb/
on “DuckDB 2.0.0 LTS “Cinnamon Teal””Submitted Super-fast deduplication of large datasets using Splink and DuckDB as a story · 4 points
HN discussionI'm working on a free high performance address matching (geocoding) library. As it turns out I blogged about it just today: https://www.robinlinacre.com/address_matching/
Comment linking /address_matching/
on “Ask HN: What Are You Working On? (June 2025)”Submitted Why Probabilistic Linkage Is More Accurate Than Fuzzy Matching for Data Deduping as a story · 4 points
HN discussionOut of the current options, I strongly agree - I even wrote a blog post! https://www.robinlinacre.com/recommend_sql/ But on the other hand, that's doesn't mean SQL is ideal - far from it. When using DuckDB with Python, to make things more succinct, reusable and maintainable, I often fall into the pattern of writing Python functions that generate SQL strings. But that hints at the drawbacks of SQL: it's mostly not co…
Comment linking /recommend_sql/
on “Python is not a great language for data science”Submitted Address matching using a fault tolerant trie as a story · 3 points
HN discussionI think these are all fair points and a good reflection of the downside of Python. But there are also some pretty huge upsides. My view is that for any model of significant complexity, the pros of Python outweigh the cons from a technical point of view. - Abstraction. It's very difficult to effectively abstract parts of a model in Excel. It's a bit like a doctor having to 'model' a human being as a collection of ato…
Comment linking /transforming_analytical_functions/
on “Ditching Excel for Python in a legacy industry”Submitted Putting Scaffolding Around Vibe Coding to Build More Complex Apps as a story · 3 points
HN discussionI don't think that's entirely fair. For instance, I host content on my personal website [0] that is written in using Observable, and in fact the site provides guides to allow anyone to export content[1] and embed elsewhere online. My content can be entirely under my control if I save my libraries out of the Observable API as a tarball[2]. I genuinely think the differences between Observable and 'pure' javascript are…
Comment linking /energy-usage/
on “Why We're Building Observable”Submitted Why DuckDB is my first choice for data processing as a story · 3 points
HN discussionAgreed. I spent some time making a calculator [0] to help me understand what matters when it comes to energy consumption. This made it very clear that by far the best way most consumers can make a difference is just to stop buying stuff. In the west we get away with 'low'(er) emissions because we import a huge amount of CO2 though imported goods - maps of total energy consumption are quite interesting here [1] Stopp…
Comment linking /energy-usage
on “We need to take CO2 out of the sky – an overview of climate tech”Submitted AI probably won't replace me in 2025 as a story · 3 points
HN discussionThe performance is pretty good because the prediction is ultimately just adding up match weights. Much of the performance is dictated by (1) The blocking approach you choose (how wide you cast the net in searching for matches. This is actually somewhere were minhash can be used in conjunction (2) whether you choose to use complex fuzzy matching functions and how many - this is chosen by the user. There's some benchm…
Comment linking /fast_deduplication/
on “Finding near-duplicates with Jaccard similarity and MinHash”Submitted The emerging impact of LLMs on my productivity as a story · 3 points
HN discussionThe Fellegi Sunter model is able to estimate the importance of different types of information from the data itself (i.e. unsupervised learning). For instance, a match on a date of birth column lends a greater weight of evidence in favour of a match than a match on first name (since dob has higher cardinality). The method is also able to estimate weights for fuzzy matches (how much evidence in favour of a match is cl…
Comment linking /fellegi_sunter_accuracy/
on “Finding near-duplicates with Jaccard similarity and MinHash”Submitted Why parquet files are my preferred API for bulk open data (2023) as a story · 2 points
HN discussionFor deduplicating people, the Fellegi Sunter model is also a powerful approach. Splink[0] is a free Python library that implements this for big datasets. Probably you could combine parts of both approaches as well. Full disclosure: I am the lead author. I've also written up some interactive tutorials on how the method works [1] if anyone's interested [0] https://github.com/moj-analytical-services/splink [1] https://…
Comment linking /intro_to_probabilistic_linkage/
on “Finding near-duplicates with Jaccard similarity and MinHash”Submitted Letterpaths – free software for teaching cursive writing as a story · 2 points
HN discussionAs a data data scientist, I also found this pretty confusing so I spent some time trying to understanding it better. I wrote it up as a blog post: Demystifying Apache Arrow https://www.robinlinacre.com/demystifying_arrow/
Comment linking /demystifying_arrow/
on “Comparing SQLite, DuckDB and Arrow with UN trade data”Submitted Building Accurate Address Matching Systems as a story · 2 points
HN discussion...which sounds totally unrealistic until you compare US energy usage to other countries: http://robinlinacre.com/country_energy_usage (Note: the above does not include the amount of energy embodied in imported goods, so if anything the energy of rich countries is even higher)
Comment linking /country_energy_usage
on “CO2 emissions set to fall nearly 8% this year, the largest drop in history”Submitted Demystifying Apache Arrow as a story · 2 points
HN discussionI'm having to adjust the energy use for heating fact some of that heat actually goes into hot water. So the calculator is figuring how much heat is needed to heat the water you use for washing, and subtracting that from the total amount on your energy bill (the remainder of which is heat). Yeah - one of the things i learned building this is the huge amount of energy it takes to move humans anywhere (in cars/planes).…
Comment linking /carbon_electric_car/
on “2000-Watt Society”Submitted Why parquet files are my preferred API for bulk open data as a story · 2 points
HN discussion