The title is a bit of an understatement because I think it's pretty good. It's not perfect and it's not guaranteed to scale, but it works pretty well. Especially on search term typos.
This, my own blog, now has a search engine built with Elasticsearch using the Python library elasticsearch-dsl. The algorithm (if you can call it that) is my own afternoon hack invention. Before I explain how it works try out a couple of searches:
Try a couple of searches:
(each search appends &debug-search for extended output)
- corn - finds Cornwall, cron, Crontabber, crontab, corp etc.
- crown - finds crown, Crowne, crowded, crowds, crowd etc.
- react - finds create-react-app, React app, etc.
- jugg - finds Jung, juggling, judging, judged etc.
- pythn - finds Python, python2.4, python2.5 etc.
Also, by default it uses Elasticsearch's match_phrase so when you search for a multi-word thing, it requires a match on each term. E.g. date format which finds Date formatting, date formats etc.
But if you search for something where the whole phrase can't match, it splits up the search an uses a match operator instead (minus any stop words).
Typo-focussed
This solution is very much focussed on typos. One thing I really dislike in non-Google search engines is when you make a search and nothing is found and it says "Did you mean ...?". Quite likely I did, but why do I have to click it? Can't it just be clicked for me?
Also, if there's ambiguity and possibly some results based on what you typed and multiple potential "Did you mean...?". Why not just blend them alltogether like Google does? Here is my attempt to solve that. Come with me...
Figuring Out ALL Search Terms
So if you type "Firefix" (not "Firefox", also scroll to the bottom to see the debug table) then maybe, that's an actual word that might be in the database. Then by using the Elasticsearch's Suggesters it figures out alternative spellings based on frequency distributions within the indexed content. This lookup is actually really fast. So now it figures out three alternative ways to spell this term:
firefox(score 0.9, 1 character different)firefli(score 0.7, 2 characters different)firfox(score 0.7, 2 characters different)
And, very arbitrarily I pick a score for the default term that the user typed in. Let's pick 1.1. Doesn't matter gravely and it's up for future tuning. The initial goal is to not bury this spelling alternative too far back.
Here's how to run the suggester for every defined doc type and generate a list of other search terms tuples (minimum score >=0.6).
search_terms = [(1.1, q)]
_search_terms = set([q])
doc_type_keys = (
(BlogItemDoc, ('title', 'text')),
(BlogCommentDoc, ('comment',)),
)
for doc_type, keys in doc_type_keys:
suggester = doc_type.search()
for key in keys:
suggester = suggester.suggest('sugg', q, term={'field': key})
suggestions = suggester.execute_suggest()
for each in suggestions.sugg:
if each.options:
for option in each.options:
if option.score >= 0.6:
better = q.replace(each['text'], option['text'])
if better not in _search_terms:
search_terms.append((
option['score'],
better,
))
_search_terms.add(better)
Eventually we get a list (once sorted) that looks like this:
search_terms = [(1.1 'firefix'), (0.9, 'firefox'), (0.7, 'firefli'), (0.7, 'firfox')]
The only reason the code sorts this by the score is in case there are crazy-many search terms. Then we might want to chop off some and only use the 5 highest scoring spelling alternatives.
Building The Boosted OR-query
In this scenario, we're searching amongst blog posts. The title is likely to be a better match than the body. If the title mentions it we probably want to favor that over those where it's only mentioned in the body.
So to build up the OR-query we'll boost the title more than the body ("text" in this example) and we'll build it up using all possible search terms and boost them based on their score. Here's the complete query.
strategy = 'match_phrase'
if original_q:
strategy = 'match'
search_term_boosts = {}
for i, (score, word) in enumerate(search_terms):
# meaning the first search_term should be boosted most
j = len(search_terms) - i
boost = 1 * j * score
boost_title = 2 * boost
search_term_boosts[word] = (boost_title, boost)
match = Q(strategy, title={
'query': word,
'boost': boost_title,
}) | Q(strategy, text={
'query': word,
'boost': boost,
})
if matcher is None:
matcher = match
else:
matcher |= match
search_query = search_query.query(matcher)
The core is that it does Q('match_phrase' title='firefix', boost=2X) | Q('match_phrase', text='firefix', boost=X).
Here's another arbitrary number. The number 2. It means that the "title" is 2 times more important than the "text".
And that's it! Now every match is scored based on how suggester's score and whether it be matched on the "title" or the "text" (or both). Elasticsearch takes care of everything else. The default is to sort by the _score as ultimately dictated by Lucene.
Match Phrase or Match
In this implementation it tries to match using a match phrase query which basically tries to find matches where every word in the query matches.
The cheap solution here is to basically keep whole search function as is, but if absolutely nothing is found with a match_phrase, and there were multiple words, then just recurse over one more time and do it with a match query instead.
This could probably be improved and do the match_phrase first with higher boost and do the match too but with a lower boost. All in one big query.
Want A Copy?
Note, this copy is quite a mess! It's a personal side-project which is an excuse for experimentation and goofing around.
The full search function is here.
Please don't judge me for the scrappiness of the code but please share your thoughts on this being a decent application of Elasticsearch for smallish datasets like a blog.
Comments