Top all-time · #771
Cloudera
Company blog · blog.cloudera.comdataai
Cloudera is the company blog at blog.cloudera.com, writing about data and ai. Since 2012, 86 different Hacker News readers have linked it 134 times, most often “Cloudera Impala: Real-Time Queries in Apache Hadoop”. No feed was found, so read it on the site itself.
Sometimes recommended · 7 of 40 sampled comments recommend it?
Why HN links it
HN readers link Cloudera's engineering blog for explainers on Hadoop, Impala, Spark, Kudu and Arrow, mostly as background or pointers to technical details.
Summarised by Claude from a sample of 40 HN comments that link here. Each comment is marked recommending, critical or neutral (citing without judging); how this works.
Cloudera Blog is your source for expert guidance on the latest data and AI trends, technology innovation, best practices, success stories, and more.
Most-linked posts?
- Cloudera Impala: Real-Time Queries in Apache Hadoop7 people · 7 mentions · first linked 2012-10-24
- Inside Cloudera Impala: Runtime Code Generation with LLVM3 people · 5 mentions · first linked 2013-02-12
- Analyze Twitter Data with Apache Hadoop (2012)3 people · 3 mentions · first linked 2013-12-06
- Feather: A Fast On-Disk Format for Data Frames for R and Python3 people · 3 mentions · first linked 2016-03-29
- /blog/2 people · 3 mentions · first linked 2015-05-19
- Using LLVM Code Generation To Speed Up Query Performance In Impala 2 people · 2 mentions · first linked 2013-02-22
- Quorum-based Journaling for the Hadoop HDFS NameNode2 people · 2 mentions · first linked 2012-10-31
- /blog/2016/02/introducing-apache-arrow-a-fast-interoperable-in-memory-columnar-data-structure-standard/2 people · 2 mentions · first linked 2016-02-18
- Hive-On-Spark Nears Production Readiness2 people · 2 mentions · first linked 2015-12-02
- /blog/2011/02/avoiding-full-gcs-in-hbase-with-memstore-local-allocation-buffers-part-1/2 people · 2 mentions · first linked 2013-01-21
What Hacker News says?
1) Put it on figshare or github. Your website might be dead next year. Those services have a better chance of survival. 2) CSV, JSON works fine if your dataset is just a few numbers and strings. GitHub will preview csv files as HTML tables. 3) If you need the efficiency of binary data and more robust data containers I would look into - Parquet https://parquet.apache.org/documentation/latest/ and - Arvo http://avro.a…
Comment linking /blog/2009/11/avro-a-new-format-for-data-interchange/
on “Google Dataset Search”Submitted Cloudera taken private for $5.3B, acquires Datacoral and Cazena as a story · 153 points
HN discussionS3 ListObjects calls are eventually consistent (i.e. list-after-put). EMRFS [1] and S3Guard [2] mitigate this for data processing use cases. [1] - https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-f... [2] - https://blog.cloudera.com/introducing-s3guard-s3-consistency...
Comment linking /introducing-s3guard-s3-consistency-for-apache-hadoop/
on “Ask HN: Is your company sticking to on-premise servers? Why?”Submitted Algorithms Every Data Scientist Should Know: Reservoir Sampling as a story · 120 points
HN discussionThe author is missing a big gap between 5TB - 1PB. For most workloads, I would not look to Hadoop at the 5TB+ scale of data. I would first look at Impala or Redshift. http://blog.cloudera.com/blog/2012/10/cloudera-impala-real-t...
Comment linking /blog/2012/10/cloudera-impala-real-time-queries-in-apache-hadoop-for-real/
on “Don't use Hadoop when your data isn't that big”Submitted A Guide to Python Frameworks for Hadoop as a story · 93 points
HN discussionDatabase optimization posts are always interesting, but it's really hard to do any apples to apples comparison. Your performance is going to depend mostly on your hardware, internal database settings and tunings, and OS level tunings. I'm glad this one included some insight into the SQLite settings disabled, but there's always going to be too many factors to easily compare this to your own setup. For most SQL system…
Comment linking /how-to-use-hbase-bulk-loading-and-why/
on “Inserting a billion rows in SQLite under a minute”Submitted Common Probability Distributions: The Data Scientist’s Crib Sheet as a story · 90 points
HN discussion(I wrote MoSQL) PostgreSQL scales surprisingly well for this purpose, and is much nicer for interactive queries than Hadoop/Hive. We use Impala[1] for some larger datasets, but Impala is comparatively new, and it's nice to have something as battle-tested as postgres here. As for the "why do we need realtime?": In my mind the benefit of a near-realtime replica is not that you actually often need it, but that it means…
Comment linking /blog/2012/10/cloudera-impala-real-time-queries-in-apache-hadoop-for-real/
on “Announcing MoSQL”Submitted BinaryPig: Malware analysis with Hadoop, Django and Elasticsearch as a story · 33 points
HN discussionCloudera | Location: Budapest, Hungary | Full-time | Onsite | Software Developers in Test (Platform Distribution), Software Developers (Platform Distribution), Solutions Architect, Customer Operations Engineers and more | www.cloudera.com Picture yourself at Cloudera in 2016! Our Budapest office is growing exponentially and we are looking for super-smart tech people to help us grow. Our mission: Founded by leading e…
Comment linking /blog/
on “Ask HN: Who is hiring? (January 2016)”Submitted Benchmarking Time Series Workloads with Kudu, InfluxDB and ClickHouse as a story · 14 points
HN discussionCloudera | Location: San Francisco or Palo Alto, CA | Full-time| Onsite | Software Engineers in Cloud, Test, Site Reliability, Hadoop, Test, Automation, Spark, QA, Security and more | www.cloudera.com Picture yourself at Cloudera in 2016! Our mission: Founded by leading experts on big data from Facebook, Google, Oracle and Yahoo, Cloudera's mission is to bring the power of Hadoop, MapReduce, and distributed storage…
Comment linking /blog/
on “Ask HN: Who is hiring? (January 2016)”Submitted Kudu: New Apache Hadoop Storage Engine as a story · 14 points
HN discussionAlso of interest: "Algorithms Every Data Scientist Should Know: Reservoir Sampling" http://blog.cloudera.com/blog/2013/04/hadoop-stratified-rand...
Comment linking /blog/2013/04/hadoop-stratified-randosampling-algorithm/
on “Algorithms Every Programmer Should Know: Reservoir Sampling”Submitted How to Resample from a Large Data Set in Parallel with R on Hadoop as a story · 14 points
HN discussionYou might find this interesting, a database query engine using LLVM IR to speed up performance. http://blog.cloudera.com/blog/2013/02/inside-cloudera-impala...
Comment linking /blog/2013/02/inside-cloudera-impala-runtime-code-generation/
on “LLVM Intermediate Representation is better than assembly”Submitted How Spark, Scala, and Functional Programming Made Hard Problems Easy at Barclays as a story · 11 points
HN discussionCloudera's Impala SQL query engine uses LLVM to maximize performance, here's an article published recently talking about how it's used. http://blog.cloudera.com/blog/2013/02/inside-cloudera-impala...
Comment linking /blog/2013/02/inside-cloudera-impala-runtime-code-generation/
on “LLVM Intermediate Representation is better than assembly”Submitted Feather: A Fast On-Disk Format for Data Frames for R and Python as a story · 9 points
HN discussionFrom what I heard, while InfluxDB is a great product it may struggle with processing big volumes of data - see https://blog.cloudera.com/benchmarking-time-series-workloads...
Comment linking /benchmarking-time-series-workloads-on-apache-kudu-using-tsbs/
on “TimescaleDB raises $40M”Submitted Wargaming.net’s Data-Driven, Real-Time Rules Engine as a story · 8 points
HN discussionIncredible piece of software. I've used it in production at my last two jobs. You can build almost anything in NiFi once you get into the mindset of how it works. A good way to get started with NiFi is to use it as a highly available quartz-cron scheduler. For example, running "some process" every 5 seconds. Disclaimer: I'm an Apache NiFi committer. An article you might find interesting about it's ability to scale.…
Comment linking /benchmarking-nifi-performance-and-scalability/
on “Apache NiFi”Submitted Cloudera Impala: Real-Time Queries in Apache Hadoop as a story · 7 points
HN discussionSMM is another option and has some really cool features for alerting as well. Runs in any cloud or on premise. https://blog.cloudera.com/smm-1-2-released-with-powerful-new...
Comment linking /smm-1-2-released-with-powerful-new-alerting-and-topic-lifecycle-management-features-with-schema-registry-integration/
on “KafkaHQ”Submitted Inside Cloudera Impala: Runtime Code Generation with LLVM as a story · 6 points
HN discussionNote that this proposal is being back not only by Google, but also Cloudera, Data Artisans, Talend, Cask, PayPal, ... Some other posts on the announcement: http://googlecloudplatform.blogspot.com/2016/01/Dataflow-and... http://blog.cloudera.com/blog/2016/01/spark-dataflow-joins-g... http://data-artisans.com/dataflow-proposed-as-apache-incubat... http://blog.cask.co/2016/01/cask-anticipates-googles-dataflo...
Comment linking /blog/2016/01/spark-dataflow-joins-googles-dataflow-sdk/
on “Google proposes its Dataflow batch/stream tech to the Apache Incubator”Submitted Apache Kudu and Apache Impala (Incubating): The Integration Roadmap as a story · 5 points
HN discussionIt depends, if you plan to scan our entire data set it could take 30-40 seconds (roughly ~2.8TB), but we have our data partitioned based on a key that makes sense for the kind of data you'd need to populate a web page and these queries are fast enough (< 2 seconds) for aggregations that come in via AJAX. We haven't yet had a chance to optimize our environment either. For example, our nodes are still running a pretty…
Comment linking /blog/2013/02/inside-cloudera-impala-runtime-code-generation/
on “AWS Redshift: How Amazon Changed The Game”Submitted Cloudera open sources distributed test running infrastructure as a story · 4 points
HN discussion
