Top all-time · #3861
Hongli Lai
Personal blog · joyfulbikeshedding.com/blogprogrammingstartups
Hongli Lai's personal blog at joyfulbikeshedding.com covers programming and startups. Since 2018, 20 different Hacker News readers have linked it 55 times, most often “Full-system dynamic tracing on Linux using eBPF and bpftrace”. It is still writing: the last post was on 2025-10-23, and you can follow it by feed.
Too few sampled comments (6) to say how often HN recommends it.?
Why HN links it
HN readers link Hongli Lai's blog for in-depth explanations of systems internals: eBPF and DTrace tracing, memory allocators such as jemalloc and malloc_trim, Python memory use, and macOS dynamic linking.
Summarised by Claude from a sample of 6 HN comments that link here. Each comment is marked recommending, critical or neutral (citing without judging); how this works.
Hongli Lai is a CTO, consultant and entrepreneur. On his website he blogs about coding, company building and other advice.
Most-linked posts?
- Full-system dynamic tracing on Linux using eBPF and bpftrace4 people · 5 mentions · first linked 2019-02-01
- A robust distributed locking algorithm based on Google Cloud Storage3 people · 7 mentions · first linked 2021-05-19
- What causes Ruby memory bloat?3 people · 6 mentions · first linked 2019-03-14
- The status of Ruby memory trimming and how you can help with testing (2019)3 people · 3 mentions · first linked 2019-03-30
- Why Fullstaq Ruby? – Joyful Bikeshedding2 people · 5 mentions · first linked 2020-05-15
- The Future of Traveling Ruby2 people · 2 mentions · first linked 2021-01-07
What Hacker News says?
I recently wrote about this subject (with sources). See these: https://www.joyfulbikeshedding.com/blog/2018-04-17-should-no... https://news.ycombinator.com/item?id=17058645
Comment linking /blog/2018-04-17-should-non-eu-websites-ban-eu-visitors-under-the-gdpr.html
on “GDPR will pop the adtech bubble”Submitted What causes Ruby memory bloat? as a story · 185 points
HN discussionI did research into the glibc memory allocator. Turns out this is not memory fragmentation, but per-thread caches that are never freed back to the kernel! A free() call does not actually free the memory externally unless in exceptional circumstances. The more threads and CPU cores you have, the worse this problem becomes. One easy solution is setting the "magic" environment variable MALLOC_ARENA_MAX=2, which limits…
Comment linking /blog/2019-03-14-what-causes-ruby-memory-bloat.html
on “Make Ubuntu packages 90% faster by rebuilding them”Submitted Full-system dynamic tracing on Linux using eBPF and bpftrace as a story · 139 points
HN discussionWhat? You can setup log rotation in 1 minute. In 3 minutes you can write a small paragraph that explains you only use IP addresses for security reasons and only store them for a few weeks. Also, the claim that you immediately get the maximum fine of 20 million euro for every small detail that you get wrong, is false: https://www.joyfulbikeshedding.com/blog/2018-04-17-should-no... (claim backed up by a book written b…
Comment linking /blog/2018-04-17-should-non-eu-websites-ban-eu-visitors-under-the-gdpr.html
on “Google and Facebook accused of breaking GDPR laws”Submitted How Debian Packaging Works as a story · 6 points
HN discussionWhat is 2021-02-28 + 1 month? Is that 2021-03-28 or 2021-03-31? It is far from obvious what a normal human would do most of the time here. The lack of associativity is still a problem. If 2021-02-28 + 1 month = 2021-03-28,then (2021-02-28 + 1 month) + 1 month = 2021-03-28 + 1 month = 2021-04-28. While if I ask what is 2021-02-28 + 2 months (given 2021-02-28 + 1 month = 2021-03-31), most people would say 2021-04-30.…
Comment linking /blog/2021-05-19-robust-distributed-locking-algorithm-based-on-google-cloud-storage.html
on “Good Design is Imperfect Design, Part 1: Honest Names”Submitted The status of Ruby memory trimming and how you can help with testing (2019) as a story · 4 points
HN discussion> Naturally I wanted to get my hands on this stuff to try it out for myself. Is it really as good as they say? Normally I stick to Linux, but it (generally) doesn’t have these Sun technologies. Since then, bpftrace[0] has been released, supposedly it's actually better than dtrace, in the sense that it's even more capable[1]. See https://www.joyfulbikeshedding.com/blog/2019-01-31-full-syst... for a recent basic ebpf/…
Comment linking /blog/2019-01-31-full-system-dynamic-tracing-on-linux-using-ebpf-and-bpftrace.html
on “Debugging Emacs, or How I Learned to Stop Worrying and Love DTrace (2018)”Submitted Studying the Kubernetes Ingress system as a story · 4 points
HN discussionI think that is an overly cautious take on things. I'll explain why. First, "for unknown reasons" deserves more nuance. The vague, high-level reason is clear: Jemalloc 3 behaves differently from Jemalloc 5, having different algorithms and data structures. What I mean by unknown is not so much an indication of incomprehensible arcane magic, and that things can collapse at any time. What I mean is that it's not known…
Comment linking /blog/2019-03-14-what-causes-ruby-memory-bloat.html
on “Fullstaq Ruby: Ruby, optimized for production”Submitted Why I avoid Prometheus+Grafana and choose Netdata for simple server monitoring as a story · 4 points
HN discussion> But I'm not convinced it's a product and not a service, or a task for someone on staff for larger organisations. From my experience with users and the industry at large, the amount of people who are comfortable with compilation and sysadmin-y tasks are rapidly declining, proportion-wise. The industry is heading towards ever-more specialization. Many many backend developers nowadays don't want to think about infras…
Comment linking /blog/2020-05-15-why-fullstaq-ruby.html
on “Fullstaq Ruby: Ruby, optimized for production”Submitted Why piping to 'cat' can stuck a CI job as a story · 3 points
HN discussion> If you're running (say) 10 Rails processes on a machine with 10 GB RAM, and particular request paths/background jobs whatever pushes the process size to 2 GB RAM size, you can release the memory back to the OS after you're done, but you still have the underlying problem which is the process gets to that size in the first place, and if all the processes hit 2GB at once, now you're 10 GB in debt, either in swap, or…
Comment linking /blog/2019-03-14-what-causes-ruby-memory-bloat.html
on “Fullstaq Ruby: Ruby, optimized for production”Submitted The status of Ruby memory trimming and how you can help with testing as a story · 3 points
HN discussionAuthor of Fullstaq Ruby here. From what I understand by talking to people, the reasons why Fullstaq Ruby doesn't have wider adoption are as follows: It's not a project officially blessed by Ruby core. Upstream Ruby is actually "good enough" for most. Most people don't feel the pain enough to try something which appears as unofficial. Perhaps their thinking are similar to yours: I'm hesitant because not everbody else…
Comment linking /blog/2020-05-15-why-fullstaq-ruby.html
on “Fullstaq Ruby: Ruby, optimized for production”Submitted Yes, team communication is supposed to be hard as a story · 3 points
HN discussionAuthor of Fullstaq Ruby here. There are absolutely no paid features, nor plans to monetize. There are even enough reasons to not monetize. See the FAQ: https://github.com/fullstaq-labs/fullstaq-ruby-server-editio... And the project vision, which is community-based: https://www.joyfulbikeshedding.com/blog/2020-05-15-why-fulls...
Comment linking /blog/2020-05-15-why-fullstaq-ruby.html
on “Fullstaq Ruby: Ruby, optimized for production”Submitted Clear Kubernetes namespace contents before deleting the namespace, or else as a story · 2 points
HN discussionhttps://www.joyfulbikeshedding.com/blog/2019-03-14-what-caus... is a pretty interesting resource about this.
Comment linking /blog/2019-03-14-what-causes-ruby-memory-bloat.html
on “Ask HN: Go programming language is over ten years old. What do you think of it?”Submitted A robust distributed locking algorithm based on Google Cloud Storage as a story · 2 points
HN discussion> At this point jemalloc is so good Some folks disagree: > [Ruby's] memory usage only reduces when using jemalloc 3; memory usage is still high when using jemalloc 5. Nobody knows why, so that makes the choice of defaulting to jemalloc very dodgy. via https://www.joyfulbikeshedding.com/blog/2019-03-29-the-statu...
Comment linking /blog/2019-03-29-the-status-of-ruby-memory-trimming-and-how-you-can-help-with-testing.html
on “Zstandard v1.4.0”Submitted Docker and the host filesystem owner matching problem as a story · 2 points
HN discussionI wrote an intro for bpftrace: https://www.joyfulbikeshedding.com/blog/2019-01-31-full-syst...
Comment linking /blog/2019-01-31-full-system-dynamic-tracing-on-linux-using-ebpf-and-bpftrace.html
on “DTrace-on-Windows”Submitted The Future of Traveling Ruby as a story · 2 points
HN discussionI blogged about this same problem a month ago. "Docker and the host filesystem owner matching problem": https://www.joyfulbikeshedding.com/blog/2021-03-15-docker-an... In my blog post I layout 2 solution strategies, how one might go about implementing them, and caveats to watch out for. 1. Matching the container's UID/GID with the host's UID/GID. 2. Remounting the host path in the container using BindFS.
Comment linking /blog/2021-03-15-docker-and-the-host-filesystem-owner-matching-problem.html
on “File Permissions: A painful side of Docker (2019)”Submitted Why Fullstaq Ruby? – Joyful Bikeshedding as a story · 2 points
HN discussionOne thing that has been known for a while is that there are allocation patterns where LibC malloc doesn't give back a sufficient amount of pages even when the pages are clean. See for example https://www.joyfulbikeshedding.com/blog/2019-03-14-what-caus... . In the end I think the consensus was that jemalloc was just better than invocing malloc_trim, but invoking malloc_trim now and then can certainly be a lot better…
Comment linking /blog/2019-03-14-what-causes-ruby-memory-bloat.html
on “Run Python Applications Efficiently with malloc_trim”Submitted A better way to reason about software testing terms as a story · 2 points
HN discussion
