Top all-time · #2606
Michael Noll
Personal blog · michael-noll.comdataprogramming
Michael Noll writes Michael G. Noll, a personal blog at michael-noll.com about data and programming. Since 2007, 32 different Hacker News readers have linked it 35 times, most often “Writing An Hadoop MapReduce Program In Python”. The last post was in 2021, but the archive is still widely read.
Most-linked posts?
- Writing An Hadoop MapReduce Program In Python6 people · 6 mentions · first linked 2011-08-04
- Writing An Hadoop MapReduce Program In Python5 people · 5 mentions · first linked 2007-10-08
- Running Hadoop On Ubuntu Linux (Single-Node Cluster) - Michael G. Noll3 people · 3 mentions · first linked 2008-06-05
- Of Algebirds, Monoids, Monads, and Other Bestiary for Large-Scale Data (2013)2 people · 3 mentions · first linked 2013-12-10
- Implementing Real-Time Trending Topics in Storm2 people · 2 mentions · first linked 2013-01-23
- /projects/spear-algorithm/2 people · 2 mentions · first linked 2011-07-27
What Hacker News says?
Hadoop itself consists of different parts: HDFS is a distributed filesystem that can span across lot's of machines and stores data in blobs of varying sizes. MapReduce is the Google idea from before 2004 how to do calculations on lot's of data. Now there is also YARN that could be described as a general job scheduler. At the moment a lot of people use software on YARK like Spark (does more in memory, is faster, can…
Comment linking /tutorials/writing-an-hadoop-mapreduce-program-in-python/
on “Deep Learning Is Easy – Learn Something Harder”Submitted What every software engineer should know about Apache Kafka as a story · 597 points
HN discussionExactly, which is why the expert information is teased out in a couple in depth blog posts. https://www.michael-noll.com/projects/ Does a very good job in this regard. I think if you have a consulting agency where you can train or lead on multiple projects, then a book could be a very good sales vehicle.
Comment linking /projects/
on “How to write a programming book”Submitted Implementing Real-Time Trending Topics in Storm as a story · 73 points
HN discussionhave you tried have you tried http://www.michael-noll.com/tutorials/running-hadoop-on-ubun... and then migrate to multi node http://www.michael-noll.com/tutorials/running-hadoop-on-ubun...
Comment linking /tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/
on “Big Data: principles and best practices (new book)”Submitted Of Algebirds, Monoids, Monads, and Other Bestiary for Large-Scale Data (2013) as a story · 66 points
HN discussionThis python API into Delicious could give you a place to start: http://www.michael-noll.com/projects/delicious-python-api/ Unfortunately Delicious will throttle you if you hit the service more often than once a second so you might not be able to get too much valuable information.
Comment linking /projects/delicious-python-api/
on “Delicious's Data Policy is Like Setting a Museum on Fire”Submitted Integrating Kafka and Spark Streaming: Code Examples and State of the Game as a story · 8 points
HN discussionIs it this one :) ? http://www.michael-noll.com/projects/spear-algorithm/
Comment linking /projects/spear-algorithm/
on “GitHire”Submitted Writing An Hadoop MapReduce Program In Python as a story · 8 points
HN discussionIt would be really nice to have dataset available for researchers with username, their posts and their comments. As an example, we could use the "SPEAR Algorithm"[1] to find expertise in specific domains per HN user. [1] http://www.michael-noll.com/projects/spear-algorithm/
Comment linking /projects/spear-algorithm/
on “Ask HN: How many HN members are there?”Submitted Writing An Hadoop MapReduce Program In Python as a story · 4 points
HN discussionThe Hadoop book gives lots of insights into the internals of Hadoop. But if you want to get started quickly you might be better off just reading a quick guide ( http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Li... ) and using the Pig console.
Comment linking /wiki/Running_Hadoop_On_Ubuntu_Linux_%28Single-Node_Cluster%29
on “O'Reilly discounts: machine learning, nltk, hadoop, sql, nosql, aws and appeng”Submitted Writing An Hadoop MapReduce Program In Python as a story · 4 points
HN discussionMichael Noll has some really good introductory articles to Big Data, and I highly recommend his posts for anyone starting in this field. His post on writing Python streaming programs in Hadoop - http://www.michael-noll.com/tutorials/writing-an-hadoop-mapr... - got me started in this area two years ago, and that's what I'm still doing these days! :)
Comment linking /tutorials/writing-an-hadoop-mapreduce-program-in-python/
on “Implementing Real-Time Trending Topics in Storm”Submitted Of Algebirds, Monoids, Monads, and Other Bestiary for Large-Scale Data Analytics as a story · 3 points
HN discussionPython: http://www.michael-noll.com/wiki/Writing_An_Hadoop_MapReduce... Ruby: http://blog.rapleaf.com/dev/?p=16 It's pretty hokey, either running through the shell or calling a REST service through a proxy, but it's been done if you're allergic to non-scripting languages.
Comment linking /wiki/Writing_An_Hadoop_MapReduce_Program_In_Python
on “Ex-Google, Yahoo, Facebook employees start company to support Hadoop”Submitted Of Streams and Tables in Kafka and Stream Processing as a story · 2 points
HN discussionHaving written MR jobs in the original Java format, I can tell you that it's quite valuable to do so before learning a DSL like Scalding or Pig. I found http://www.michael-noll.com/tutorials/running-hadoop-on-ubun... to be a very good tutorial for a quickstart guide to Hadoop.
Comment linking /tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/
on “Why Hadoop MapReduce needs Scala”Submitted Understanding the Internal Message Buffers of Storm (2013) as a story · 2 points
HN discussionI found this article to be cool https://www.michael-noll.com/blog/2013/12/02/twitter-algebir...
Comment linking /blog/2013/12/02/twitter-algebird-monoid-monad-for-large-scala-data-analytics/
on “Ask HN: Best “Big Data” Course?”Submitted Writing an Hadoop MapReduce Program in Python (2012) as a story · 2 points
HN discussionThere isn't any pain. Best of all it's easy to setup hadoop on a single machine, or two machines here and there and then them into a large cluster: Here's a good tutorial that may help: http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Li... (If you're using a Linux machine just follow that; otherwise get VMWare running Ubuntu and set it up there).
Comment linking /wiki/Running_Hadoop_On_Ubuntu_Linux_%28Single-Node_Cluster%29
on “Hadoop use at Facebook ”Submitted Writing An Hadoop MapReduce Program In Python Michael G. Noll as a story · 2 points
HN discussionJava's not actually a bad choice, considering some of the tools that are available. Hadoop[1], HBase[2], etc. Depending on what you are trying to accomplish of course, you could also include Lucene[3], Solr[4], Mahout[5] and Weka[6] in that list. Of course cool Java libraries don't mean you have to work in Java. Scala[7] seems to be gaining quite a bit of momentum, as does Clojure[8]. From your reading list, it soun…
Comment linking /wiki/Writing_An_Hadoop_MapReduce_Program_In_Python
on “Best language to learn for Big Data”Submitted Writing An Hadoop MapReduce Program In Python as a story · 2 points
HN discussionWhat If We Could Rebuild Kafka From Scratch? - Gunnar Morling ( https://news.ycombinator.com/item?id=43790420 ) 1. Five Advantages of Log-Based Change Data Capture ( https://debezium.io/blog/2018/07/19/advantages-of-log-based-... ) 2. Why Data Architecture Should Guide Your Query Engine Decision ( https://www.onehouse.ai/blog/why-data-architecture-should-gu... ) 3. Reimagine Apache Kafka ( https://streamnative.io/bl…
Comment linking /blog/2018/04/05/of-stream-and-tables-in-kafka-and-stream-processing-part1/
on “Show HN: Discover and link the indieweb with a simple text query”Submitted Of Algebirds, Monoids, Monads, and Other Bestiary for Large-Scale Data Analytics as a story · 1 points
HN discussionThere're many things you can do with a spare linux box. I do keep a few 'project ideas' links just in case I got some spare time on hand. Home file server with ZFS - http://breden.org.uk/2008/03/02/a-home-fileserver-using-zfs/ Proxy Server - http://devcentral.f5.com/weblogs/macvittie/archive/2008/10/0... Map-Reduce fun with Hadoop - http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Li... Donate bandwidth wit…
Comment linking /wiki/Running_Hadoop_On_Ubuntu_Linux_(Single-Node_Cluster)
on “Ask HN: What to do with a spare linux box?”Submitted Realtime Trending Topics With A Distributed Rolling Count Algorithm in Storm as a story · 1 points
HN discussion
