Top all-time · #2683
Changing Bits
Personal blog · blog.mikemccandless.comprogrammingdata
Changing Bits is a personal blog at blog.mikemccandless.com about programming and data. Since 2011, 28 different Hacker News readers have linked it 30 times, most often “Lucene's FuzzyQuery is 100 times faster in 4.0”. The last post was in 2021, but the archive is still widely read.
Most-linked posts?
- Lucene's FuzzyQuery is 100 times faster in 4.011 people · 11 mentions · first linked 2011-04-27
- How we added merge-on-refresh to Apache Lucene3 people · 3 mentions · first linked 2021-03-26
- Changing Bits: Lucene index in RAM with Azul's Zing JVM3 people · 3 mentions · first linked 2012-08-01
What Hacker News says?
Jason from Elastic here. A shard can handle concurrent writes from clients, there are multiple write threads scaling to the number of logical processors exposed to Elasticsearch, and we have invested in making Lucene accommodate this concurrency[0][1][2]. [0]: http://blog.mikemccandless.com/2011/05/265-indexing-speedup-... [1]: https://issues.apache.org/jira/browse/LUCENE-3023 [2]: http://blog.mikemccandless.com/201…
Comment linking /2017/07/lucene-gets-concurrent-deletes-and.html
on “Guide to running Elasticsearch in production”Submitted Lucene's FuzzyQuery is 100 times faster in 4.0 as a story · 92 points
HN discussionPeople with an interest may want to check out Azul's Zing JVM and the following blog post testing it out with Lucene... http://blog.mikemccandless.com/2012/07/lucene-index-in-ram-w...
Comment linking /2012/07/lucene-index-in-ram-with-azuls-zing-jvm.html
on “Open-sourcing a 10x reduction in Apache Cassandra tail latency”Submitted Language detection with Google's Compact Language Detector as a story · 75 points
HN discussionWell, Solr and Lucene projects are right now in the process of splitting up (after joining for versions 3-8). And, you could have always used Lucene directly as an embedded search library. Amazon does, for example, for their customer facing search as their scalability patterns do not align with either Solr or Elasticsearch. And if you do use Lucene directly, you can choose just the libraries that apply to your use c…
Comment linking /2021/03/open-source-collaboration-or-how-we.html
on “Building a full-text search engine in 150 lines of Python code”Submitted Lucene's FuzzyQuery is 100 times faster in 4.0 (2011) as a story · 48 points
HN discussionUnknown is relative, but here are some of the rss feeds I'm subscribed too: http://nosql.mypopescu.com/ http://lambda-the-ultimate.org/ http://eli.thegreenplace.net/ http://www.stefankrause.net/wp/ http://mechanical-sympathy.blogspot.com/ http://www.cowtowncoder.com/blog/blog.html http://highlyscalable.wordpress.com/ http://hunch.net/ http://perspectives.mvdirona.com/ http://www.scottaaronson.com/blog/ http://www.so…
Comment linking /
on “Ask HN: What unknown technical blogs or sites do you read?”Submitted How we added merge-on-refresh to Apache Lucene as a story · 3 points
HN discussionIt has long since been cleaned up, but the history of Lucene's Levenshtein automatons are a fascinating mix of pragmatism, hackery, grit, and engineering. Using a Python library to generate crazy-looking Java code, which you can't (yet) understand, but whose tests pass and benchmarks look amazing. - https://blog.mikemccandless.com/2011/03/lucenes-fuzzyquery-i... - https://www.youtube.com/watch?v=4AbcvHKX8Ow (Levensh…
Comment linking /2011/03/lucenes-fuzzyquery-is-100-times-faster.html
on “Ask HN: What piece of code/codebase blew your mind when you saw it?”Submitted Apache Lucene 7.0 Is Coming Soon as a story · 3 points
HN discussionA footnote: For Java devs out there, the Lucene codebase is a joy to read [0]. Esp, the APIs. Highly recommend it just for the documentation which is top-notch. Michael McCandless, Lucene's committer-in-chief, sometimes blogs about its internals [1]. [0] https://github.com/apache/lucene-solr/tree/master/lucene [1] http://blog.mikemccandless.com/2019/10/concurrent-query-exec...
Comment linking /2019/10/concurrent-query-execution-in-apache.html
on “Elasticsearch from the Bottom Up (2013)”Submitted (2011)Lucene's FuzzyQuery is 100 times faster in 4.0 as a story · 2 points
HN discussionLucene is quite fantastic and Elasticsearch makes it a joy to use. Still, I wonder what the overhead of Java is adding in this case. Even minor things like integer decoding can be done very fast with SIMD... but such approaches don't seem amenable to Java. I see that Elasticsearch exposes quite a bit of GC metrics, which must be a problem at times. And one of the Lucene devs wrote a post on how he replaced some part…
Comment linking /2011/03/lucenes-fuzzyquery-is-100-times-faster.html
on “Lucene: The Good Parts”Submitted Changing Bits: Lucene index in RAM with Azul's Zing JVM as a story · 2 points
HN discussionAlso interesting, the cutting edge work lucene has done to make this possible in 4.x, using universal automatons to speed up fuzzy, regex, and other searches. http://blog.mikemccandless.com/2011/03/lucenes-fuzzyquery-is...
Comment linking /2011/03/lucenes-fuzzyquery-is-100-times-faster.html
on “Fast and easy Levenshtein distance using a trie (2011)”Submitted Lucene's FuzzyQuery is 100 times faster in 4.0 (2011) as a story · 1 points
HN discussionIf anyone wants to know how deep down the rabbit hole this stuff goes, they should read this blog post [1] on writing a Levenshtein Automaton to speed up fuzzy matching in Lucene by 100 times. It gets deep! [1]: http://blog.mikemccandless.com/2011/03/lucenes-fuzzyquery-is...
Comment linking /2011/03/lucenes-fuzzyquery-is-100-times-faster.html
on “Generating all permutations, combinations, and power set of a string (2012)”Submitted Open-source collaboration, or how we added merge-on-refresh to Apache Lucene as a story · 1 points
HN discussionThe Lucene search library has something like this. Data is stored /retrieved via codecs. Most write binary data, but one codec actually writes out all data as text. It is, of course, too slow for production use, but it is great for figuring out what is going on under the covers. https://blog.mikemccandless.com/2010/10/lucenes-simpletext-c...
Comment linking /2010/10/lucenes-simpletext-codec.html?m=1
on “Enter a command to see help text for each arg”Submitted Lucene's FuzzyQuery is 100 times faster in 4.0 (2011) as a story · 1 points
HN discussionMike McCandless gives a good overview here: http://blog.mikemccandless.com/2012/07/lucene-400-alpha-at-l... He has some other posts on there that do a deeper dive as well.
Comment linking /2012/07/lucene-400-alpha-at-long-last.html
on “Apache releases Lucene Core 4.0 and Solr 4.0”Submitted Changing Bits: Choosing a fast unique identifier (UUID) for Lucene as a story · 1 points
HN discussionIt's OK to not like Solr. At the same time, half of the features I listed above, are quite new (SolrCloud, docValues, LTR, Config and Schema API, JSON support, etc). And the other half - the 'old' part you may not like - are battle-tested, multiple-times speed-optimized pieces of code. Like this one: http://blog.mikemccandless.com/2011/03/lucenes-fuzzyquery-is... Moreover, their architecture makes very clear that th…
Comment linking /2011/03/lucenes-fuzzyquery-is-100-times-faster.html
on “Architecture of Nautilus, the new Dropbox search engine”Submitted Changing Bits: Lucene with Zing as a story · 1 points
HN discussionThere's a quite amusing blog post about implementing the automaton for Lucene: http://blog.mikemccandless.com/2011/03/lucenes-fuzzyquery-is... As an aside, I know one of the authors of the original paper personally. Quite interesting guy, and the remark about the paper being "nearly unintelligible" is fitting.
Comment linking /2011/03/lucenes-fuzzyquery-is-100-times-faster.html
on “The Levenshtein distance in production (2020)”This blog post [ http://blog.mikemccandless.com/2012/08/lucenes-new-blockpost... ] and the associated Lucene Jira issue [ https://issues.apache.org/jira/browse/LUCENE-3892 ] may be of interest to those who are excited by this paper. If I'm reading it right, I think the main conclusion from the Lucene patch was that using a straight "FOR" (Frame of Reference) approach was faster in the real world than using "PFOR" (P…
Comment linking /2012/08/lucenes-new-blockpostingsformat-thanks.html
on “Fast integer compression: decoding billions of integers per second”Something of a story at http://blog.mikemccandless.com/2011/03/lucenes-fuzzyquery-is...
Comment linking /2011/03/lucenes-fuzzyquery-is-100-times-faster.html
on “Levenshtein Automata (2010)”To add another one: Lucene uses a finite state transducer to represent a part of its term index: http://blog.mikemccandless.com/2010/12/using-finite-state-tr... --- It's essentially a dictionary as you say, but instead of a direct NLP use case, it might map a word to some numerical value (like a file offset where the term's postings list resides). I actually haven't been able to find anyone else using transducers fo…
Comment linking /2010/12/using-finite-state-transducers-in.html
on “Finite State Machines in Clojure”I believe we're somewhat constrained by our license with Oracle regarding what we can publish for benchmarks. But others are not, and this piece by Mike McCandless gives some good insight into what's possible with Zing: http://blog.mikemccandless.com/2012/07/lucene-index-in-ram-w...
Comment linking /2012/07/lucene-index-in-ram-with-azuls-zing-jvm.html
on “Azul JVM Now Available For Developers of Open Source Applications”
