Top all-time · #3198
The Astronomer Blog: Data Orchestration Insights & Guides
Company blog · astronomer.io/blog · on HN as rywalkerdataprogramming
The Astronomer Blog: Data Orchestration Insights & Guides is the company blog at astronomer.io, writing about data and programming. Since 2015, 24 different Hacker News readers have linked it 100 times, most often “Apache Spark Streaming for Real-Time Customer Insights”. It is still writing: the last post was on 2026-10-15, and you can follow it by feed.
Too few sampled comments (26) to say how often HN recommends it.?
Why HN links it
HN readers link Astronomer's blog for technical Apache Airflow and data orchestration guides, such as the task-groups and Airflow 2.0 articles. One commenter says its technical pieces are on point.
Summarised by Claude from a sample of 26 HN comments that link here. Each comment is marked recommending, critical or neutral (citing without judging); how this works.
Apache Airflow® orchestrates the world’s data, ML, and AI pipelines. Astro is the best way to build, run, and observe them at scale.
Most-linked posts?
- /3 people · 6 mentions · first linked 2018-02-04
- /3 people · 3 mentions · first linked 2017-09-14
- /2 people · 5 mentions · first linked 2020-03-13
- /airflow2 people · 2 mentions · first linked 2018-02-23
- Apache Spark Streaming for Real-Time Customer Insights2 people · 2 mentions · first linked 2017-08-03
- What's Coming in Apache Airflow 2.02 people · 2 mentions · first linked 2020-10-30
- Machines and Humans = Future of SaaS2 people · 2 mentions · first linked 2016-08-18
- Why We Built Our Data Platform on AWS, and Why We Rebuilt It with Open Source2 people · 2 mentions · first linked 2016-07-22
What Hacker News says?
I love SMIL animations. If you're curious about what you can do, here's a few examples: https://www.pcmaffey.com/roll-your-own-analytics https://www.pcmaffey.com/finally-i-closed-my-linkedin https://www.astronomer.io/404.html (let this play > 1 minute) You can open any of these images and View Source to dive in to the implementation... except for Edge, SMIL plays exactly the same in all browsers, whereas I've found…
Comment linking /404.html
on “Where are all the animated SVGs?”Submitted An Almost Acquisition Story as a story · 81 points
HN discussionhttp://www.astronomer.io/blog all of their technical pieces are pretty on point
Comment linking /blog
on “Ask HN: Which companies have the best blogs written by their engineering team?”Submitted Why Astronomer is Betting on GraphQL as a story · 70 points
HN discussionastronomer.io | Everything | Fully Remote | Full Time We're the commercial developers behind Apache Airflow! If you are looking to build on a K8s native stack around one of the most popular open source projects around, please shoot us a note! https://www.astronomer.io/careers/
Comment linking /careers/
on “Ask HN: Who is hiring? (April 2021)”Submitted Python to Distributed Python to Airflow task in ~5 lines of code as a story · 35 points
HN discussionSide gig with COSS companies https://coss.media https://astronomer.io https://hofstadter.io https://toptal.com
Comment linking /
on “Ask HN: Dear open source devs how do you sustain yourself”Submitted Using Apache Airflow to Create Data Infrastructure in the Public Sector as a story · 8 points
HN discussionWe use airflow mainly for ETL work, backfills, and batch processing. We do a lot of work with clickstream type data, be it taking data from analytics.js and loading it into redshift, or taking analytic data from redshift and loading into Google analytics. We have developed many pipes and connectors for airflow that allow us to connect to many data sources, both at the source and sink ends. I mainly work on the DevOp…
Comment linking /
on “S3 was down”Submitted Five Ways to Make Sure Your Analytics Spark Growth as a story · 6 points
HN discussion1) An official Helm chart is coming very soon. We (Astronomer) have a commercial platform that aims to solve this completely including observability, configuration and deployment. Happy to team up to improve Airflow's open-source k8s story if you have some ideas. 2) Yes the UI is outdated, and not responsive. We're going to kick off a process to build a new modern UI in Q3 (a full-featured Swagger API is being built…
Comment linking /blog/the-keda-autoscaler/
on “Introduction to Apache Airflow”Submitted Why We Built Our Data Platform on AWS, and Why We Rebuilt It with Open Source as a story · 6 points
HN discussionAstronomer | Senior Software Engineer | US REMOTE | Full-time | https://www.astronomer.io/ At Astronomer, our R&D team is dedicated to providing an exceptional experience in managing Apache Airflow at scale. As a leading player in the industry, we are seeking two experienced Software Engineers to work on our data observability product - Astro Observe. For more details and to apply: https://jobs.ashbyhq.com/astronome…
Comment linking /
on “Ask HN: Who is hiring? (January 2025)”Submitted Astronomer Airflow v0.2.0 – A fully open source alternative to Google Composer as a story · 5 points
HN discussionAstronomer | https://astronomer.io | Remote | Full-Time Astronomer is the commercial developer of Apache Airflow, a community-driven open-source tool that’s leading the market in data orchestration. We’re a globally-distributed and rapidly growing venture-backed team of learners, innovators and collaborators. Our mission is to build an Enterprise-grade product that makes it easy for data teams at Fortune 500’s and s…
Comment linking /
on “Ask HN: Who is hiring? (December 2021)”Submitted Lessons Learned Writing Data Pipelines as a story · 5 points
HN discussionAt Airbnb, we were using SubDAGs to try to manage large number of tasks in a single DAG. This allowed organizing tasks and drilling down into failures more easily but came with its own challenges. In more recent versions of Airflow, TaskGroups ( https://airflow.apache.org/docs/apache-airflow/stable/concep... , https://www.astronomer.io/guides/task-groups/ ) were made to help this a little bit. Hopefully that helps a…
Comment linking /guides/task-groups/
on “Lessons learned from running Apache Airflow at scale”Submitted Six Open Source Dashboards as a story · 5 points
HN discussionAstronomer | https://astronomer.io | NYC preferably, open to remote | Full-time Hiring for ~10 eng roles listed at astronomer.io/careers, but the focus is specifically on distributed systems, observability, and applied AI. Astronomer is a primary contributor to Apache Airflow and offers our customers a managed Airflow service with a similar business model as Databricks with Spark or Confluent with Kafka. Our tech st…
Comment linking /blog/astro-airflow-re-engineered-for-speed-and-scale
on “Ask HN: Who is hiring? (September 2026)”Submitted Orchestrating Data Quality with Airflow as a story · 4 points
HN discussionAirflow is extremely active — check the mailing list, Gitter chat, or even GitHub contributions. My company, Astronomer, also provides a hosted version of Airflow [1] if SaaS is more you're thing. [1]: https://www.astronomer.io/airflow
Comment linking /airflow
on “Retiring Wave”Submitted Build LLM Workflows Before Agents as a story · 4 points
HN discussionWe're using Apache Airflow [2] internally as well. It's pretty featureful and addresses most concerns mentioned by OP, such as orchestration, retries, open source code reuse, and dependency management. It has primitive monitoring and alerting, but one needs to bring something external for that today. Shameless plug - My startup [2] offers Airflow as a SaaS as well as an enterprise distribution with monitoring tools…
Comment linking /
on “Ask HN: How do you handle long-running workflows at your company?”Submitted Write Airflow DAGs using vanilla Python and SQL, notebook style as a story · 4 points
HN discussionDBT in simplest terms is just a way to orchestrate transformations in dependency order. Table "A" needs to be updated before table "B," because "B" selects from "A." It works well for that. I've seen it used to get wild west SQL logic into version control. To replace scheduled SQL workbooks running entire data pipelines. I've also seen over-engineered integrations with other orchestration tools in the "stack." Once…
Comment linking /cosmos/
on “Is the "modern data stack" still a useful idea?”Submitted Announcing Astronomer v0.9, a Platform to Run Apache Airflow on Kubernetes as a story · 4 points
HN discussionWe've outlined some of the key new features of Airflow 2.0 here - https://www.astronomer.io/airflow - and it's available on Astronomer Cloud today.
Comment linking /airflow
on “Airflow 2.0”Submitted On Universal Music Group's Airflow Leak as a story · 4 points
HN discussionHere's an article from the same source: https://www.astronomer.io/blog/introducing-airflow-2-0/
Comment linking /blog/introducing-airflow-2-0/
on “AWS Managed Workflows for Apache Airflow”Submitted Apache Spark Streaming for Real-Time Customer Insights as a story · 4 points
HN discussion
