For our most recent hack day, I worked with my colleagues Antonio Barone and Nelio Nunes to implement an alert-me-when function for our site. In production, we use Solr at the moment as the search engine to serve our frontend. Unfortunately, implementing alerting functionality with it means you have to go down a do-it-yourself route.
So, for our hack, we used Elasticsearch, which has a really nice built-in feature called Percolator that is exactly what we needed. Borrowing the percolation definition from the Search::Elasticsearch::Client::Direct CPAN module:
“Percolation is search inverted: instead of finding docs which match a particular query, it finds queries which match a particular document.”
Since we wanted to present the final result with some real data, we needed to switch the frontend to use Elasticsearch instead of Solr, hence we needed to import data into Elasticsearch from Solr. That was easily achieved using River, a plugin for Elasticsearch to import Solr data. (The details of the migration are off-topic for this post.)
The Percolate queries are just normal documents with a specific format, so they can be decorated with any extra information and queried as any other index. The query documents are stored in an arbitrary index under the reserved type name .percolator.
Here are somes snippets of code from our implementation…the code is part of a Dancer app.
- Check if the search return results otherwise store the query
- Check if the new product matches any query and send the email
- Elasticsearch offers scalable alert-me-when functionality out of the box with the Percolator feature; doing the same in Solr would have required writing it ourselves.
- The perl module provide a good interface to Elasticsearch, but there are still some rough edges around the Percolator interface. Mainly it looks to be a read-only interface, so in order to add queries to the percolator index requires the use of the standard interface. This ends up exposing some Elasticsearch internals that would be nice to mask behind the percolator interface.