Top all-time · #21601
GlassFlow
Company blog · glassflow.dev/blogdataprogramming
GlassFlow is the company blog at glassflow.dev, writing about data and programming. Since 2025, 3 different Hacker News readers have linked it 9 times. No feed was found, so read it on the site itself.
Too few sampled comments (2) to say how often HN recommends it.?
Why HN links it
HN readers link GlassFlow's posts on streaming data pipelines, mainly Kafka to ClickHouse and removing duplicates. Comments ask how its deduplication works and say the blog lacked detail.
Summarised by Claude from a sample of 2 HN comments that link here. Each comment is marked recommending, critical or neutral (citing without judging); how this works.
Give AI agents the context they need and observe their actions. Built for both short & always-on, long-running agents. Open source, self-hosted.
What Hacker News says?
How does the deduplication itself work? The blog didn't have many details. I'm curious because it's no small feat to do scalable deduplication in any system. You have to worry about network latencies if your deduplication mechanism is not on localhost, the partitioning/sharding of data in the source streams, and handling failures writing to the destination successfully, all of which cripples throughput. I helped mai…
Comment linking /blog/Part-5-How-GlassFlow-will-solve-Duplications-and-JOINs-for-ClickHouse
on “Show HN: GlassFlow – OSS streaming dedup and joins from Kafka to ClickHouse”Submitted Load Test GlassFlow for ClickHouse: Real-Time Dedup at Scale as a story · 22 points
HN discussionYes its true that if you just want to send data from Kafka to clickhouse and do not worry about duplicates, then there are several ways. we even covered them in a blog post -> https://www.glassflow.dev/blog/part-1-kafka-to-clickhouse-da... However, the reason for us to start building this was because duplication is a sad reality in streaming pipelines and the methods to clean up duplicates on clickhouse is not good…
Comment linking /blog/part-1-kafka-to-clickhouse-data-ingestion
on “Show HN: GlassFlow – OSS streaming dedup and joins from Kafka to ClickHouse”Submitted ClickHouse Denormalization is not the answer to slow JOINs as a story · 18 points
HN discussionHi HN, A few weeks ago, we shared GlassFlow: Open Source streaming ETL to dedup and join streams from Kafka for ClickHouse ( https://news.ycombinator.com/item?id=43953722 ). One of the top questions we received was: “How well does it perform at high throughput?” We ran a load test and would like to share some results with you. Summary of the test: - Tested on 20m records - Kafka produced 55,000 records/sec - Process…
Comment linking /blog/load-test-glass-flow-for-click-house-real-time-deduplication-at-scale
on “Load Test GlassFlow for ClickHouse: Real-Time Dedup at Scale”Submitted Limits of ClickHouse ReplacingMergeTree and Materialized Views for Data Streams as a story · 3 points
HN discussionI am seeing this pattern a lot lately. Teams start with a simple flow: logs/metrics → Vector → ClickHouse Works well as long as they run simple transformations via Vector. When they start adding things like dedupe, longer time windows, more data volume or joins, things start to break. They actually start using Vector as a stream processing engine. Very typical issue that I see: Time window limits: By default vector…
Comment linking /blog/when-vector-becomes-your-streaming-engine
on a deleted threadSubmitted Kafka to ClickHouse Ingestion Methods as a story · 2 points
HN discussionSubmitted ClickHouse ReplacingMergeTree doesn't work for streaming data as a story · 1 points
HN discussion