Top all-time · #4366
engineering.pivotal.io
Company blog · engineering.pivotal.ioprogrammingsystems
engineering.pivotal.io is the company blog at engineering.pivotal.io, writing about programming and systems. Since 2015, 18 different Hacker News readers have linked it 24 times, most often “Abstraction, Or, the Gift of Our Weak Brains”. No feed was found, so read it on the site itself.
Too few sampled comments (12) to say how often HN recommends it.?
Why HN links it
HN comments cite this Pivotal engineering blog as a source for cloud and infrastructure topics: AWS vs GCP storage benchmarks, Linux memory overcommit and swap, Cloud Foundry buildpacks and BOSH load testing, and blog publishing. No comment judges the writing itself.
Summarised by Claude from a sample of 12 HN comments that link here. Each comment is marked recommending, critical or neutral (citing without judging); how this works.
Most-linked posts?
- /post/virtual_memory_settings_in_linux_-_the_problem_with_overcommit/2 people · 2 mentions · first linked 2017-12-20
- Abstraction, Or, the Gift of Our Weak Brains2 people · 2 mentions · first linked 2015-12-19
- GPORCA, a Modular Query Optimizer, Is Now Open-Source2 people · 2 mentions · first linked 2016-01-28
What Hacker News says?
I think that basically the disk format change was too far a bridge to cross. You can't tell customers that they should quadruple their cluster so that they can dump their data and reimport it to perform the upgrade. The planner is basically entirely different. It was later extracted into a standalone module to build HAWQ[0]. Since then there has been work to build a new shared query planner called GPORCA[1]. [0] htt…
Comment linking /post/gporca-open-source/
on “Microsoft acquires Citus Data (YC S11)”Submitted Abstraction, Or, the Gift of Our Weak Brains as a story · 68 points
HN discussionMy experience at Pivotal (now VMware) echoes Dan Luu's point: the less friction to publish, the more posts on the engineering blog [0]. Our current set-up is simple: clone a repo, create a blog post (in Markdown) from a template, and when you're ready to publish, commit & push to the master branch. We put a lot of effort into making the process simple for engineers. However, before that process was put in place, it…
Comment linking /
on “How some good corporate engineering blogs are written”Submitted ETL Journey from Oracle to Postgres as a story · 4 points
HN discussionCounterexample (somewhat): http://engineering.pivotal.io/post/ntp-costs-500/#adequate
Comment linking /post/ntp-costs-500/
on “Explain like I’m 5: Kerberos (2013)”Submitted Getting Windows Containers Working in Cloud Foundry as a story · 3 points
HN discussionI've also benchmarked GCP vs. AWS [0], and, for the tests that I ran, found that GCP outperformed AWS by a factor of 3:1. Specifically, a GCP instance n1-highcpu-8 with a 256GB pd-ssd disk, clocked in at 11,728 IOPS vs an AWS c4.xlarge with a 256GB gp2 disk, clocking in at 3,634 IOPS. To put that in context of the blog post, it means your setup can drastically affect your results. Using local NVMe disk, for example,…
Comment linking /post/gobonniego_results/
on “AWS Outperforms GCP in the 2018 Cloud Report”Submitted Trilogy – the database testing framework as a story · 3 points
HN discussion> Keeping build packs up to date, keeping all the scripts running, and making sure an app has associated dependencies, etc--I imagine that's the difference between an independent setup you can self-host quickly and easily and one that's very dependent on an ecosystem of Heroku maintainers, tooling and existing server infrastructure... I work for Pivotal on the Cloud Foundry buildpacks team. 4 of our buildpacks (Ruby…
Comment linking /post/creating-a-custom-buildpack/
on “Decommissioning Otto”Submitted Abstraction, Or, the Gift of Our Weak Brains as a story · 3 points
HN discussionIt depends on whether the system you're using overcommits memory. Some OSes (eg Linux) provide settings to limit virtual memory use to some percentage of physical memory. See http://engineering.pivotal.io/post/virtual_memory_settings_i... The only time you should avoid checking malloc() return values are for special mallocs that really cannot return NULL. One example is FreeBSD's kernel malloc() when called with the…
Comment linking /post/virtual_memory_settings_in_linux_-_the_problem_with_overcommit/
on “Reading the NSA’s codebase: LemonGraph review”Submitted How to Set Up a Distributed Elixir Cluster on Amazon EC2 as a story · 2 points
HN discussionNot a perfect choice of terms I freely admit. In my own experience, I last used non-virtual, mutable, long-lived servers over 8 years ago. Since then it's been a progression further away from that style: PaaS, immutable infrastructure, Heroku, and currently a mix of AWS Lambda and Docker. I'm excited to try Elixir but from what I've read (e.g. https://medium.com/mint-digital/elixir-deployments-on-aws-ee... , http://…
Comment linking /post/how-to-set-up-an-elixir-cluster-on-amazon-ec2/
on “Ask HN: What language/framework do you plan to use in your next project?”Submitted Greenplum merge with PostgreSQL 8.3 as a story · 2 points
HN discussion> I don't understand why SQL on LLVM? SQL is mostly "do the same thing a billion times", which aligns very closely with generating code for the inner loop, particularly when the inner loop can be generated without branches. SQL codegen is basically an industry standard for large-scale sql query processing on top of a large-scale column store. I work one of the few engines at this scale which does not do go query sta…
Comment linking /post/codegen-gpdb-qx/
on “Snel: SQL Native Execution for LLVM”Submitted GPORCA, a Modular Query Optimizer, Is Now Open-Source as a story · 2 points
HN discussionAFAIK, there are only a handful of DBMSs that do complete query compilation with the LLVM: * MemSQL ( http://highscalability.com/blog/2016/9/7/code-generation-the... ) * Tableau/TUM HyPer ( https://blog.acolyer.org/2016/05/23/efficiently-compiling-ef... ) * CMU Peloton ( http://db.cs.cmu.edu/papers/2017/p1-menon.pdf ) * Vitesse ( https://www.youtube.com/watch?v=PEmVuYjhQFo ) I think Greenplum was talking about doing…
Comment linking /post/orca-profiling/
on “JITing PostgreSQL using LLVM [pdf]”Submitted Troubleshooting obscure OpenSSH failures due to a broken firewall as a story · 1 points
HN discussionI did some benchmarks of AWS vs GCP storage [0] (NOT including local storage), and the short version is that Google's slow storage is slower than AWS's, and Google's fast storage is faster than AWS's. And I tested NVMe, but on vSphere, and it was very, very fast. 0: http://engineering.pivotal.io/post/gobonniego_results/
Comment linking /post/gobonniego_results/
on “EC2 Instance Update – C5 Instances with Local NVMe Storage”Submitted How I Learned to Stop Worrying and Like Red Junit Tests as a story · 1 points
HN discussionI googled that and was pleasantly surprised to learn something new.[1] But if I'm reading that right I would probably want to do both. Because overcommit settings seem to be a memory saving technique. To avoid needing swap by decreasing the allowed overcommit memory. Anyways, in the many years that I've been changing vm.swappiness I've never had any issues and I've always had a minimum of 8G RAM so I don't think thi…
Comment linking /post/virtual_memory_settings_in_linux_-_the_problem_with_overcommit/
on “Linux Performance: Why You Should Almost Always Add Swap Space”Submitted Intro to the Patch Command as a story · 1 points
HN discussionFor those familiar with BOSH, Pivotal engineers built "Tornado" for load testing. It's a BOSH release of JMeter with some convenient dials. Since it's BOSH, you can deploy to AWS, GCP and Azure without fuss. The difference between running one worker, ten workers or a thousand workers is a line of YAML and how much you're prepared to pay. There are some blog posts: "Half a Million Concurrent/Distributed JMeter Users…
Comment linking /post/jmeter-bosh-release/
on “Distributed Performance Testing in the Cloud with JMeter and AWS”
