Top all-time · #1449
Airbyte
Company blog · airbyte.com/blogdataprogramming
Airbyte is the company blog at airbyte.com, writing about data and programming. Since 2022, 50 different Hacker News readers have linked it 135 times, most often “Will Rust take over Data Engineering?”. No feed was found, so read it on the site itself.
Rarely recommended · 1 of 32 sampled comments recommend it?
Why HN links it
HN readers link Airbyte's blog as supporting references on data integration, ETL/ELT, data lakehouses, orchestration and data engineering careers. Most comments only cite or link the posts.
- Recommended for · 1 comments
- One commenter says the linked post goes in-depth on a declarative approach to data orchestration.1
Summarised by Claude from a sample of 32 HN comments that link here. Each comment is marked recommending, critical or neutral (citing without judging); how this works.
Move data from 700+ sources without writing code. Airbyte runs in your VPC or on Airbyte Cloud and turns your warehouse into the context layer for AI agents.
Most-linked posts?
- Will Rust take over Data Engineering?5 people · 5 mentions · first linked 2022-10-21
- The Shift from Data Pipelines to Data Products4 people · 6 mentions · first linked 2022-06-14
- Challenges of Build ETLs4 people · 4 mentions · first linked 2022-02-25
- Show HN: Airbyte Cloud – ELT platform with open-source data connectors4 people · 4 mentions · first linked 2022-04-06
- Pandas 2.0 and Its Ecosystem (Arrow, Polars, DuckDB)3 people · 3 mentions · first linked 2023-03-06
- SQL vs. Python for Data Analysis2 people · 4 mentions · first linked 2022-03-17
- Data teams are not worthless2 people · 3 mentions · first linked 2023-02-28
- Data Lake / Lakehouse Guide: Data Lake Table Formats (Delta Lake, Iceberg, Hudi)2 people · 3 mentions · first linked 2022-08-25
- The Road to GA: Understanding Airbyte Connector Release Stages2 people · 2 mentions · first linked 2023-01-25
- Scaling Data Pipelines on Kubernetes2 people · 2 mentions · first linked 2022-01-06
What Hacker News says?
It is of course true... it is well known that SQL Server scales to department level, but Oracle scales to company level. This is true inside Microsoft and Oracle as well. Inside Microsoft, they have a bug database per division but Oracle has a single database for the entire company. Ask people who work at those companies. See also scalability sections in these artcles: https://airbyte.com/data-engineering-resources/…
Comment linking /data-engineering-resources/oracle-vs-sql-server
on “Databases in 2024: A Year in Review”Submitted A Technical Dive into PostgreSQL's replication mechanisms as a story · 210 points
HN discussionYes, Airbyte is using Temporal. Here is a blog post they wrote a few weeks ago that goes into more detail about it: https://airbyte.com/blog/scale-workflow-orchestration-with-t...
Comment linking /blog/scale-workflow-orchestration-with-temporal
on “Lessons learned from running Apache Airflow at scale”Submitted The evolution of the data engineer role as a story · 167 points
HN discussionFascinating article and to the heart of my being as a DE engineer. I had the same conclusion a while back and wrote about the trends in data orchestration towards a declarative pipeline, which more modern tools, as you mentioned, will do, but I'm afraid not Airflow. I'd say data consumers, such as data analysts and business users, care primarily about the production of data assets. On the other hand, data engineers…
Comment linking /blog/data-orchestration-trends
on “Airflow's Problem”Submitted Data Integration Guide: Techniques, Technologies, and Tools as a story · 57 points
HN discussionCheckout Airbyte and singer.io for data movement. They're tailored for moving offsite data local, but also have support for moving data between different database vendors. I'd recommend starting with Airbyte. For transformations I'm a sucker for dbt-core, which executes transformations as SQL against your data warehouse vs. a separate process. This keeps everything simple and (generally) snappy. It uses the Python J…
Comment linking /connectors
on “We built our customer data warehouse all on Postgres”Submitted Airbyte 1.0 – Marketplace, AI Assist, Gen AI Support and Enterprise GA as a story · 50 points
HN discussion(OP's coworker) We actually published a guide on data lakes/lakehouses last month! https://airbyte.com/blog/data-lake-lakehouse-guide-powered-b... covering: - What’s a Data Lake and Why Do You Need One? - What’s the Differences between a Data Lake, Data Warehouse, and Data Lakehouse - Components of a Data Lake - Data Lake Trends in the Market - How to Turn Your Data Lake into a Lakehouse
Comment linking /blog/data-lake-lakehouse-guide-powered-by-table-formats-delta-lake-iceberg-hudi
on “The evolution of the data engineer role”Submitted Show HN: Chat with your data using LangChain, Pinecone, and Airbyte as a story · 32 points
HN discussionI love the article, and it gave me the essential pointers to start with. I loved it so much that it inspired me to write a piece focusing on python vs. rust in data engineering. I leave the article here, in case of interest: https://airbyte.com/blog/rust-for-data-engineering
Comment linking /blog/rust-for-data-engineering
on “Carefully exploring Rust as a Python developer”Submitted Pull data from 100s of sources in 1 Python statement with in-memory ELT library as a story · 24 points
HN discussionNeither, just an interesting factoid that highlights just how zany the VC landscape was in the MDS space just a few years ago. https://airbyte.com/blog/the-deck-we-used-to-raise-our-150m-...
Comment linking /blog/the-deck-we-used-to-raise-our-150m-series-b
on “Is the "modern data stack" still a useful idea?”Submitted Navigating the Data Engineering Landscape in 2024 as a story · 19 points
HN discussionyup! here's how we think about the data integration problem. https://airbyte.com/blog/data-integration
Comment linking /blog/data-integration
on “$5.6B cloud company Fivetran acquired its way to survival”Submitted ELTP: Extending ELT for Modern AI and Analytics as a story · 16 points
HN discussionfor what its worth, we've been offering compute based pricing at Airbyte, and recently published our learnings here https://airbyte.com/blog/introducing-volume-based-pricing does this match the size-based pricing you are thinking about?
Comment linking /blog/introducing-volume-based-pricing
on “$5.6B cloud company Fivetran acquired its way to survival”Submitted Airbyte's Spring Release with a Preview of the Connector Builder AI as a story · 13 points
HN discussionSince Pandas 2, Apache Arrow replaced NumPy as the backend for Pandas. Arrow is also used by Polars, DuckDB, Ibis, the list goes on. https://arrow.apache.org/overview/ Apache Arrow solves most discussed problems, such as improving speed, interoperability, and data types, especially for strings. For example, the new string[pyarrow] column type is around 3.5 times more efficient. [...] The significant achievement here…
Comment linking /blog/pandas-2-0-ecosystem-arrow-polars-duckdb
on “Query Your Python Lists”Submitted Will Rust take over Data Engineering? as a story · 12 points
HN discussionI think its still "coming soon" https://airbyte.com/blog/airbyte-strategy-to-commoditize-all...
Comment linking /blog/airbyte-strategy-to-commoditize-all-data-integration
on “Launch HN: Castled Data (YC W22) – Open-Source Reverse ETL”Submitted The Shift from Data Pipelines to Data Products as a story · 10 points
HN discussiondo you already have an ETL pipeline to BQ? You could use Airbyte[0] for pg to bq. It's pretty heavy if this is your only source, but should work well if you need to query the pg data in bq frequently. 0 - https://airbyte.com/connections/PostgreSQL-to-BigQuery
Comment linking /connections/PostgreSQL-to-BigQuery
on “Faster PostgresSQL to BigQuery Transfers”Submitted Airbyte made progress on Postgres replication performance as a story · 8 points
HN discussionLooks super polished. Will have to give it a spin. what we use right now is Airbyte https://airbyte.com/ - super exhaustive in connectors but configuring some connectors like SFDC was very painful
Comment linking /
on “Show HN: Cube – open-source headless BI”Submitted Airbyte API & Terraform Provider – available in open source as a story · 8 points
HN discussionWhen it says "available in open source", is that under the main airbyte repo's licensing [1], hence primarily licensed under the Elastic License v2 and therefore not typically considered open source by many? Airbyte has previous of advertising their offering as open source while not really being as per the OSD[2]. This has been raised with them previously but without response [3][4]. They've also been extending thei…
Comment linking /blog/update-on-airbytes-license
on “Airbyte API & Terraform Provider – available in open source”Submitted Why Postgres is the most popular connector as a story · 7 points
HN discussion- https://hevodata.com/ - https://airbyte.com/ - https://www.stitchdata.com/ Unfortunately none of them are perfect as your data scales, each have up and downsides...
Comment linking /
on “All the ways to capture changes in Postgres”Submitted Airbyte makes 100 alpha / beta connectors free on Airbyte Cloud as a story · 6 points
HN discussion
