Top all-time · #4599
Coiled
Company blog · coiled.io/blogdataprogramming
Coiled is the company blog at coiled.io, writing about data and programming. Since 2020, 17 different Hacker News readers have linked it 28 times, most often “Advantages of Parquet File Format for Dask Analyses”. No feed was found, so read it on the site itself.
Too few sampled comments (8) to say how often HN recommends it.?
Why HN links it
HN commenters link Coiled's blog posts on Python data tooling: formatting notebooks with Black, Dask at terabyte scale, and Dask use in practice. None of the comments judge the posts directly.
Summarised by Claude from a sample of 8 HN comments that link here. Each comment is marked recommending, critical or neutral (citing without judging); how this works.
Coiled is a lightweight compute platform for Python data engineers and scientists. Scale your workloads to thousands of machines in the cloud without Docker or Kubernetes. Process terabytes of data with familiar tools like Pandas, Xarray, and Prefect.
Most-linked posts?
- Advantages of Parquet File Format for Dask Analyses2 people · 2 mentions · first linked 2022-05-04
- Dask as a Spark Replacement2 people · 2 mentions · first linked 2021-10-04
What Hacker News says?
I like Black for normal Python code, but it seems to mangle Pandas / Dask code at times. I still use it extensively cause it doesn't seem like there are other good alternatives. I wrote a blog post on how to use Black in Jupyter Lab notebooks if anyone is interested: https://coiled.io/blog/code-formatting-jupyter-notebooks-wit... It's really nice to format a notebook with the click of a button.
Comment linking /blog/code-formatting-jupyter-notebooks-with-black/
on “Black, the uncompromising Python code formatter, is stable”Submitted Seven Stages of Open Software as a story · 91 points
HN discussionCoiled | Remote-first | Full-time | Cloud Infrastructure & Product Engineers | https://coiled.io/jobs/ Coiled connects data scientists and researchers to distributed infrastructure using the Python data science stack (Numpy, Pandas, Scikit-Learn, etc) and Dask, a popular open source library for parallel analytics. We come from the open source Python community, and are long-time maintainers of this stack in general a…
Comment linking /jobs/
on “Ask HN: Who is hiring? (December 2020)”Submitted Coiled Computing as a story · 41 points
HN discussionWe use dask heavily, along with rest of the pydata ecosystem. I guess we are in the 'sweet spot' where data doesn't fit memory, to begin with, but once we perform any filtering and aggregations, switch over to pandas. That's exactly what dask recommends too. Our datasets don't exceed 100GB right now. Also note, dask clearly acknowledges challenges dealing with data in the terabyte range https://coiled.io/blog/dask-a…
Comment linking /blog/dask-as-a-spark-replacement/
on “Dask – A flexible library for parallel computing in Python”Submitted How to Run a Stack Overflow Sprint as a story · 3 points
HN discussionHere are the main advantages of Parquet over CSV (post author made these points more abstractly): * Parquet stores schema in the metadata, so schema inference isn't required (schema inference is expensive for big datasets) * Parquet files are columnar so individual columns can be grabbed for analyses (Spark does this automatically). This is a huge performance improvement. * Row groups contain min/max info for each c…
Comment linking /parquet-column-pruning-predicate-pushdown/
on “Time to retire the CSV?”Submitted Slow is smooth, smooth is fast: reducing memory usage in Dask workloads by 80% as a story · 2 points
HN discussionParquet is a wonderful file format and is a dream to work with compared to CSV. Parquet embeds the schema in the footer metadata, so the query engines don't need to guess what the column names / data types are. Parquet used to be poorly supported, but now it's well supported by almost all languages. You can even view Parquet files in text editors now, but that's not something I've ever needed ( https://blog.jetbrain…
Comment linking /blog/parquet-file-column-pruning-predicate-pushdown/
on “Why isn’t there a decent file format for tabular data?”Submitted Automate ETL Jobs in the Cloud with GitHub Actions as a story · 2 points
HN discussionHere are the big tips I think the article missed: Use the new string dtype that requires way less memory, see this video: https://youtu.be/_zoPmQ6J1aE . object types are really memory hungry and this new type is a game changer. Use Parquet and leverage column pruning. `usecols` doesn't leverage column pruning. You need to use columnar file formats and specify the `columns` argument with `read_parquet`. You can never…
Comment linking /blog/parquet-column-pruning-predicate-pushdown/
on “Tips for saving memory with Pandas”Submitted PyData Global 2021: Top 5 Highlights as a story · 2 points
HN discussionHave you taken a look at the "Zero to JupyterHub with Kubernetes" guide? https://zero-to-jupyterhub.readthedocs.io/en/latest/index.ht... Alternatively, you could check out the QHub project from Quansight. https://qhub.dev/ I think these are less relevant to you since you already have in-house k8s resources, but Coiled and Saturn Cloud offer services that might be interesting to other people with similar goals but wh…
Comment linking /
on “JupyterHub in Kubernetes”Submitted Why I Joined Coiled as a story · 2 points
HN discussionNot OP, but in my experience: Jupiter Intelligence ( https://jupiterintel.com/ ) Descartes Labs ( https://descarteslabs.com/ ) Microsoft Planetary Computer ( https://planetarycomputer.microsoft.com/ ) Coiled (managed Dask - python HPC) ( https://www.coiled.io/ ) CarbonPlan ( https://carbonplan.org/ ) Salo Sciences (acquired by Planet, satellite imagery company) ( https://salo.ai/ ) lots of others
Comment linking /
on “Request for Startups: Climate Tech”Submitted Spark to Dask: The Good, Bad, and Ugly of Moving from Spark to Dask(2022) as a story · 1 points
HN discussionSubmitted How Popular is Matplotlib? (usage on ArXiv) as a story · 1 points
HN discussionSubmitted Prioritizing Pragmatic Performance for Dask as a story · 1 points
HN discussionSubmitted Someone moved the NYC Taxi Cab data as a story · 1 points
HN discussionSubmitted Advantages of Parquet File Format for Dask Analyses as a story · 1 points
HN discussionSubmitted Apple M1: setting up PyData libraries with arm-64 native conda as a story · 1 points
HN discussionSubmitted Better Distributed Shuffling with Dask as a story · 1 points
HN discussionSubmitted Dask as a Spark Replacement as a story · 1 points
HN discussionSubmitted Scaling automated data workflows to the cloud as a story · 1 points
HN discussionSubmitted Accelerating the Dask Scheduler as a story · 1 points
HN discussionSubmitted Coiled, a Dask company, is giving all users 200 free cores for a month as a story · 1 points
HN discussion
