Top all-time · #10933
Vaex.io: An ML Ready Fast DataFrame for Python 🚀
Company blog · vaex.io/blogdataprogramming
Vaex.io: An ML Ready Fast DataFrame for Python 🚀 is the company blog at vaex.io, writing about data and programming. Since 2021, 7 different Hacker News readers have linked it 9 times. No feed was found, so read it on the site itself.
Too few sampled comments (6) to say how often HN recommends it.?
Why HN links it
HN readers link Vaex, a Python DataFrame library for out-of-core and larger-than-memory data, as an alternative to pandas and Dask. Commenters discuss it in threads on Dask, Arrow and DataFrames.
Summarised by Claude from a sample of 6 HN comments that link here. Each comment is marked recommending, critical or neutral (citing without judging); how this works.
Out-of-Core DataFrames for Python, ML, visualize and explore big tabular data at a billion rows per second 🚀 | Pandas alternative
Most-linked posts?
- /docs/index.html2 people · 2 mentions · first linked 2021-11-18
What Hacker News says?
I was mainly looking at Dask for the purpose of a more better Pandas DataFrame, but it seems like it's actually not that much better, just more distributed. It's like Dask is trying to do too many things and there doesn't do most of them well enough. While I haven't tried it, vaex[1] seems much better, since it specifically to only address one pain point, which is to address the inefficiencies of the pandas DataFram…
Comment linking /docs/index.html
on “Dask – A flexible library for parallel computing in Python”Submitted A Hybrid Apache Arrow/Numpy DataFrame with Vaex Version 4.0 as a story · 58 points
HN discussionFWIW, I'm moving towards [vaex]( https://vaex.io/docs/ ) for out of core python stuff Dask is well built but is fundamentally flawed by tacking together pandas structures with multiprocessing, rather than having a fundamentally efficient underlying structure.
Comment linking /docs/
on “Dask – A flexible library for parallel computing in Python”Submitted How to Train and Deploy a Machine ML with Vaex on Google Cloud Platform as a story · 6 points
HN discussionHi, how does this compare to Vaex? https://vaex.io/docs/tutorial.html Vaex can join larger than memory datasets, but it doesn't support SQL per-se, it uses it's own lazy dataframe DSL.
Comment linking /docs/tutorial.html
on “Apache Arrow Datafusion 5.0.0 release”Submitted Dask vs. Vaex – A Qualitative Comparison as a story · 3 points
HN discussionThanks, glad you find Vaex useful. Indeed, for small data there is not much to gain, at least this is not the focus of this article. Although even with small amounts of the, the automatic pipelines are useful https://vaex.io/blog/ml-impossible-train-a-1-billion-sample-...
Comment linking /blog/ml-impossible-train-a-1-billion-sample-model-in-20-minutes-with-vaex-and-scikit-learn-on-your
on “A Hybrid Apache Arrow/Numpy DataFrame with Vaex Version 4.0”Why this hybrid dataframe? [1]: > [Arrow] adoption will take time, and most people are probably more comfortable seeing NumPy arrays. Therefore a Vaex version 4 a DataFrame can hold both NumPy arrays and Apache Arrow arrays to make the transition period easier. There seems to be agreement that Apache Arrow is the future of dataframes across ML ecosystems. I didn't realize this transition impacted NumPy arrays in add…
Comment linking /blog/a-hybrid-apache-arrow-numpy-dataframe-with-vaex-version-4
on “A Hybrid Apache Arrow/Numpy DataFrame with Vaex Version 4.0”There are many packages that can do that, like Vax [1] and Dask [2]. I don't know exactly your workflow. But the concurrency in python is limited to multiprocessing, which is much expensive than threads which usually a typical streaming parallelizer will use outside python world. [1] https://vaex.io/docs/index.html [2] https://docs.dask.org/en/latest/
Comment linking /docs/index.html
on “Why HTTP/3 is eating the world”