Top all-time · #14244
Marc Seeger
Personal blog · blog.marc-seeger.deprogrammingsystems
Marc Seeger writes Marc's Blog, a personal blog at blog.marc-seeger.de about programming and systems. Since 2009, 5 different Hacker News readers have linked it 21 times. The last post was in 2015, but the archive is still widely read.
Too few sampled comments (1) to say how often HN recommends it.?
Why HN links it
Linked for a practical overview of key-value stores and interfaces for Ruby. The only comment is a bare link to a PDF on large-site architecture, so the writing itself is not discussed.
Summarised by Claude from a sample of 1 HN comments that link here. Each comment is marked recommending, critical or neutral (citing without judging); how this works.
What Hacker News says?
My Master's degree project was a webcrawler. If you're already reading this, the thesis[0] might be a somewhat entertaining read. I had a bit different constraints (only hitting frontpage, cms/webserver/... fingerprinting, backend has to be able to do ad-hoc queries for site features), but it's nice to see that the process is always somewhat the same. One of the most interesting things I experienced was, that link c…
Comment linking /2010/12/09/my-thesis-building-blocks-of-a-scalable-webcrawler/
on “How to crawl a quarter billion webpages in 40 hours”Submitted Building blocks of a scalable webcrawler. as a story · 80 points
HN discussionI recently found an interesting DNS based service that will allow you to access some of them: http://blog.marc-seeger.de/2011/04/07/hulu-and-iplayer-outsi... The advantage over a VPN-based solution is that there is no need to route all of your traffic over the VPN. Most of the time they only redirect the geo-location stuff and once you receive the videostream URL, that transfer will go over your 'regular' connection…
Comment linking /2011/04/07/hulu-and-iplayer-outside-the-us-without-a-vpn
on “Why I get my entertainment via BitTorrent”Submitted Key-Value Stores: a practical overview, and interfaces for Ruby as a story · 12 points
HN discussionRelated: I also wrote a little script that uses the API to save data as YAML files: http://blog.marc-seeger.de/2012/09/16/backing-up-fitbit-data... It isn't as detailed (e.g. distribution over time) as the website, but it still gives me a warm and fuzzy feeling to have at least part of the data in machine-readable files :)
Comment linking /2012/09/16/backing-up-fitbit-data-using-their-api/
on “Fitbit, why can’t I have my data?”Submitted SRE Interviews in Silicon Valley as a story · 6 points
HN discussionI'm always amazed about how there isn't more competition in this space. As my Master's thesis [0], I built a crawler that did similar fingerprinting (although less generic). It wasn't something breathtakingly novel, but all in all a somewhat successful project. It detected > 100 CMS, additional features like ad networks, social embeds, CDN, industry detection, company size etc. In the end, you could run a search and…
Comment linking /2010/12/09/my-thesis-building-blocks-of-a-scalable-webcrawler/
on “How We Found All of Optimizely's Clients”Submitted Impressed by "Maglev" Ruby Implementation as a story · 1 points
HN discussionOh nice. I've been doing a lot of crawling myself ( http://blog.marc-seeger.de/2010/12/09/my-thesis-building-blo... ) and I'd love to get my hands on this data. I hope they'll segment their data a bit further. I personally would LOVE to have a simple list of the domainnames themselves without all of the connections and documents. Also: Why not just use bittorrent to distribute it?
Comment linking /2010/12/09/my-thesis-building-blocks-of-a-scalable-webcrawler/
on “Free 5 Billion Page Web Index Now Available from Common Crawl Foundation”Not that I know of. I usually use "unblock us" ( http://blog.marc-seeger.de/2011/04/07/hulu-and-iplayer-outsi... ), but I guess that's mainly for TV series rather than movies.
Comment linking /2011/04/07/hulu-and-iplayer-outside-the-us-without-a-vpn/
on “Bitcoin value falls from above $33 to below $3”For people interested in the more "anonymous" parts of filesharing, I wrote my Bachelor's thesis on the subject: http://blog.marc-seeger.de/2008/07/23/the-current-state-of-a... It's a bit dated at this point, but still give an ok overview of networks like Gnutella, EDonkey and Bittorrent and also goes into the more exotic ones. As far as I see it, the currently 'usable' clients that use darknets and turtle-hopping a…
Comment linking /2008/07/23/the-current-state-of-anonymous-file-sharing/
on “Tribler Makes BitTorrent Impossible to Shut Down”I personally have the WD MyBook live ( http://blog.marc-seeger.de/2011/03/20/the-western-digital-my... ), it has an 800 MHz APM82181 (PPC Architecture) with 256 MB RAM, that should run most headless file sync stuff :)
Comment linking /2011/03/20/the-western-digital-mybook-live/
on “AeroFS (YC S10): It’s Like Dropbox, But It Runs Inside Your Firewall”It's not a real "startup" type of project, but these are some of my half-finished projects that I'd like to bring to a stable and releasable point some time: ----------------- At http://blog.marc-seeger.de/ , I'm trying to get a blog based on serious ( http://github.com/colszowka/serious ) (ruby, sinatra + flat files) up and running. I don't seem to find the time (or muse) to keep doing some improvements such as: -…
Comment linking /
on “Ask HN: Show me your Half Baked project”In case somebody wants an OSX binary but doesn't want to download the development tools and qt: http://blog.marc-seeger.de/2011/01/26/phantomjs_osx_binary (Indirection over my blog in case I need to switch the file away from my webspace)
Comment linking /2011/01/26/phantomjs_osx_binary
on “PhantomJS is a minimalistic, headless, WebKit-based, JavaScript-driven tool”I recently noticed that you could create your own "IP -> countrycode" lookup system by using the delegation tables provided by arin/ripencc/afrinic/... It might not be very accurate, but it's a fun little exercise: http://blog.marc-seeger.de/2013/09/07/ip-to-countrycode-with...
Comment linking /2013/09/07/ip-to-countrycode-with-ruby/
on “Show HN: IP Lookup”I guess this fits in here: Once upon a time I wrote my thesis on building a web crawler. The (tiny) blog post with an embedded preview: http://blog.marc-seeger.de/2010/12/09/my-thesis-building-blo... The PDF itself: http://blog.marc-seeger.de/assets/papers/thesis_seeger-build... It's mostly a "this is what I learned and the things I had to take into consideration" with a few "this is how you identify a CMS" bits spr…
Comment linking /assets/papers/thesis_seeger-building_blocks_of_a_scalable_webcrawler.pdf
on “Crawling Billions of Pages: Building Large Scale Crawling Cluster, part 1”This only seems to set a proxy which is ok, but limited to your browser. Stand-alone mediaplayers (apple tv, roku, ...) won't be able to use this easily. I always liked the DNS based services[0] seeing as you can enter them in your DSL router and all wifi appliances will be able to take advantage of them. The other option I used was using socks and proxychains[1] to force e.g. XBMC through an SSH connection to a che…
Comment linking /2012/02/26/socks-and-proxychains-as-a-vpn-alternative/
on “This service lets you use Netflix, Pandora, Hulu, and Rdio from any country”direct link to pdf: http://blog.marc-seeger.de/assets/papers/Ultra_Large_Sites_S...
Comment linking /assets/papers/Ultra_Large_Sites_SS09-Seeger_Key_Value_Stores.pdf
on “Key-Value Stores: a practical overview, and interfaces for Ruby”I have them running on a WD MyBook Live [0]. It also features an 800 MHz ARM. I does only have 256 MB RAM, but does a decent job so far. Couchpotato/Sickbeard + sabnzbd + transmission run fine. Timemachine backups also used to work fine, at the moment I have a little bit of a problem with them. That might be due to me hacking arround on the system too much though :) [0] http://blog.marc-seeger.de/2011/03/20/the-west…
Comment linking /2011/03/20/the-western-digital-mybook-live/
on a deleted thread
