Top all-time · #3699
research.neustar.biz
Company blog · research.neustar.bizdatasystems
research.neustar.biz is the company blog at research.neustar.biz, writing about data and systems. Since 2014, 21 different Hacker News readers have linked it 24 times, most often “Passenger Privacy in the NYC Taxicab Dataset”. No feed was found, so read it on the site itself.
Too few sampled comments (16) to say how often HN recommends it.?
Why HN links it
Neustar Research's blog on probabilistic counting and sketches (HyperLogLog, MinHash, PCSA), differential privacy, and analyses of NYC taxi data. Commenters point readers to its explanations.
Summarised by Claude from a sample of 16 HN comments that link here. Each comment is marked recommending, critical or neutral (citing without judging); how this works.
Most-linked posts?
- Passenger Privacy in the NYC Taxicab Dataset7 people · 7 mentions · first linked 2014-09-16
- Estimating quantiles with O(1) space2 people · 2 mentions · first linked 2014-05-16
- HyperLogLog – Cornerstone of a Big Data Infrastructure2 people · 2 mentions · first linked 2014-05-16
What Hacker News says?
Only if you can/do meaningfully anonymize the data. Eg, knowing the subnet of most/all Norwegian isps, it's trivial to recover ips that are simply hashed (probably even with salt), similarly Norwegian phone numbers are only eight digits, so any kind of deterministic mapping is likely to be too trivial to actually amount to anonymization. Also rember that one of the goals is to avoid illicit linking - so being able t…
Comment linking /2014/09/15/riding-with-the-stars-passenger-privacy-in-the-nyc-taxicab-dataset/
on “How do I permanently delete my account?”Submitted Passenger Privacy in the NYC Taxicab Dataset as a story · 132 points
HN discussionOne of the ways we count distinct elements in an unknown amount of data--probabilistically! You may have heard of Bloom filters and the like, and this is a similar family of algorithms. https://research.neustar.biz/2012/07/09/sketch-of-the-day-k-...
Comment linking /2012/07/09/sketch-of-the-day-k-minimum-values/
on “Books I used to develop a game engine”Submitted HyperLogLog – Cornerstone of a Big Data Infrastructure as a story · 63 points
HN discussionThis article mixes up a couple different issues. First, internal controls: what can employees do when they access the user database? Beginning startups probably don't have a lot of formal policies and you're relying on good judgement of the founders and first employees. As a company grows, they need to guard user data with access restrictions, restrict root to a few trusted sysadmins, and put in place clear rules ab…
Comment linking /2014/09/15/riding-with-the-stars-passenger-privacy-in-the-nyc-taxicab-dataset/
on “We Can't Trust Uber”Submitted Writing Analytics SQL with Common Table Expressions as a story · 57 points
HN discussionIt is a bit unsettling that this information is out there, but I agree that it is fairly obvious that they have this data. And as long as they are not exposing the individuals, I don't see it as irresponsible to publish something like this. There was a related story published recently, NYC Taxicab Dataset Exposes Strip Club Johns and Celebrity Trips http://research.neustar.biz/2014/09/15/riding-with-the-stars...
Comment linking /2014/09/15/riding-with-the-stars-passenger-privacy-in-the-nyc-taxicab-dataset/
on “Rides of Glory – Uber Blog (2012)”Submitted Differential Privacy: The Basics as a story · 3 points
HN discussionNeustar Research has a good article on this which may be of interest to you. However, I think the approach they took was still based on the inclusion-exclusion principle (still a good read though). * https://research.neustar.biz/2012/12/17/hll-intersections-2/ Another approach if you have the flexibility of changing your implementation, is to use a different type of data-sketch that is more amenable to set expressio…
Comment linking /2012/12/17/hll-intersections-2/
on “Show HN: An improved version of HyperLogLog”Submitted Estimating quantiles with O(1) space as a story · 2 points
HN discussion>Are there any major ones I'm missing, or even better, any lesser known ones I should be aware of? MinHash. I also highly recommend this blog: https://research.neustar.biz/tag/sketching/
Comment linking /tag/sketching/
on “Show HN: Cuckoo Filter Implementation in Go, Better Than Bloom Filters”Submitted An Entity Resolution Primer as a story · 1 points
HN discussionI am extremely glad that most comments here see the inherent craziness of this idea. I am a hardware designer who participated heavily in mining since Jan 2011, and gave up completely on Bitcoin in summer of 2014. The big problem that Balaji is completely ignoring is that SHA256 cores implemented in silicon have extraordinarily high power consumption relative to their size (mm2). This is due to the incredibly high s…
Comment linking /2012/02/02/choosing-a-good-hash-function-part-3/
on “A Bitcoin miner in every hand”Submitted Frugal Streaming as a story · 1 points
HN discussionYou can probably trust Uber just about as much as you can trust the TLC. The only real difference is that Uber has a bit more real--time data they can mine. Even if you only pay for your cabs in cash, your privacy can be compromised by pick up/drop off point meta data ( http://research.neustar.biz/2014/09/15/riding-with-the-stars... ). Then what about all the other plethora of apps and OS features that do background…
Comment linking /2014/09/15/riding-with-the-stars-passenger-privacy-in-the-nyc-taxicab-dataset/
on “Can We Trust Uber?”Another good explanation, this one with a javascript simulation of HLL: http://research.neustar.biz/2012/10/25/sketch-of-the-day-hyp...
Comment linking /2012/10/25/sketch-of-the-day-hyperloglog-cornerstone-of-a-big-data-infrastructure/
on “How a HyperLogLog works”If you dig through to the source reddit posting [1], you can see that the post only really puts 6 different individual taxi IDs out there for visualization. To me, only the first 4 seem like a good visual fit for the prayer times, and the overall trip heatmap [2] suggests that they may just be part of a larger pattern of eating/taking a break at sunrise, noon, and sunset for all cab drivers. While the blog posting i…
Comment linking /2014/09/15/riding-with-the-stars-passenger-privacy-in-the-nyc-taxicab-dataset/
on “Using NYC Taxi Data to Identify Muslim Taxi Drivers”@Rurban (Reini Urban): wrt to your first point, general purpose hash functions being modeled as PRNGs, I believe that statement to be correct. It is definitely the case when analysing hash function behaviour in contexts such as bloom filters, hash tables etc. With regards to the definition of perfect hash functions - if you had read the sentence before the one you have quoted, you would have notice that the formal d…
Comment linking /tag/hash-function/
on “Brief Introduction To General Purpose Hash Functions”I'm not able to review the data yet but I wonder if TLC took in any of the feedback to better anonymize the data. [0] Also, I would love to see an analysis of whether traffic is actually getting worse compared to that of last year. This claim was made by mayor de Blasio as a reason to cap Uber rides. [0] http://research.neustar.biz/2014/09/15/riding-with-the-stars...
Comment linking /2014/09/15/riding-with-the-stars-passenger-privacy-in-the-nyc-taxicab-dataset/
on “NYC's Taxi and Limousine Commission Trip Record Data for 2014-2015”HyperLogLog is a really interesting and powerful data type. Though a few things seem missed in the article. HyperLogLog has been around I believe over 10 years now. It's been supported for Postgres for over 5 years [1] now as an extension[2]. It is great to see it growing further with Facebook adding support for it for Presto. [1]. https://research.neustar.biz/2012/10/25/sketch-of-the-day-hy... [2]. https://www.citu…
Comment linking /2012/10/25/sketch-of-the-day-hyperloglog-cornerstone-of-a-big-data-infrastructure/
on “HyperLogLog in Presto: Faster cardinality estimation”Streaming algorithms, or sketches, are a fascinating topic. Some of the results are virtually indistinguishable from magic. I can recommend this blog for those interested: http://research.neustar.biz/tag/sketching/
Comment linking /tag/sketching/
on “HyperLogLog and MinHash”Probabilistic Counting with Stochastic Averaging : https://research.neustar.biz/2013/04/02/sketch-of-the-day-pr...
Comment linking /2013/04/02/sketch-of-the-day-probabilistic-counting-with-stochastic-averaging-pcsa/
on “Show HN: Reducing the size of PCSA to that of HyperLogLog”To anyone interested by differential privacy I would recommend this article: http://research.neustar.biz/2014/09/08/differential-privacy-... It does a very good job of explaining the basics. They also published a follow up with the quite popular taxi dataset: http://research.neustar.biz/2014/09/15/riding-with-the-stars...
Comment linking /2014/09/15/riding-with-the-stars-passenger-privacy-in-the-nyc-taxicab-dataset/
on “Differential privacy for dummies”
