IDOL To Solr Migration Lessons Learned - Part 3 Data Processing

This post is a quick follow up to the last one in which I discussed several differences between IDOL and Solr when it comes to indexing data. In this post we will explore how different products maintain the data and what they allow you to do with it.

With IDOL were able to set certain fields for various searches, such as IndexFields for text indexing, or parametric for other types of searches. Any kind of data massaging had to happen before indexing to IDOL; via custom development or a Lua script in CFS. This certainly got the job done, but it didn’t even come close to the flexibility that Solr offers. With Solr, we customized processing for each field using Solr’s built in analyzers and tokenizes. This treats each field independently and tokenize (create individual searchable elements) by field. After tokenization, we run several analyzers on fields. Analyzers allow you to massage the data a bit before it is indexed; things like stemming, synonyms, minimum token length filtering occurs at this stem.

For example, we have a filed for manufacturer part number, the data comes in a variety of different formats and no single rule would accommodate all manufacturers and their favorite preference for part number formats. However, we were able to take it up a notch, but using the Word Delimiter Filter (Analyzer) to massage a bit each token in the manufacturer part number field. The word delimiter filter is configured to break up a token at transition from numeric to characters and vice versa. This breaks up patters such as FLV12345X, then, the length filter is applied to filter out anything less than two characters, so this field would be indexed as FLV12345X (original term is kept), FLV and 12345.  If a user searches for a part number as it is indexed, there will be a hit on three terms in one field, all of these terms should hit. Since the field is weighted highly, that almost guarantees that the product will be displayed. This came in handy, since we don’t have control over our data and get even the same part numbers in different formats from each vendor.

An even cool part, is that you can specify two sets of processes for each field, index and query level processing. This allows you to run a specific process on data when you are indexing it, and then a slightly different process when you are querying it. This came in handy for synonyms as we didn’t want to expand the query to contain synonyms, since some of them are abbreviations and they could dilute the search results, so instead we just included them at index time and indexed a few extra terms for each picked up synonym.

Solr’s list of tokenizes and analyzers can be found here: https://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters

Don’t worry if the list looks intimidating at first, Solr’s admin interface lets you test how various strings will look like with different field configurations. This allows you to easily test any field level processing without having to drum up some dummy content, index it, and then search for it until you figure out what is what. The Analyzer tab in Solr UI lets you see exactly what will happen with your data when you index it.

To sum up and keep it short: Solr is definitely a winner in this category. IDOL has some other cool functionality, especially when dealing with conceptual understanding of various terms, but it's not the same as this awesomeness of Solr field level processing.

I hope you found this post informative, I think that field processing is by far the coolest thing about Solr; but there are several others that are pretty close behind it. If you liked this post and want to get future posts by me, remember to sign up in the top right.

IDOL To Solr Migration Lessons Learned - Part 2 Indexing Data

Since the data we were indexing into IDOL was plain old XML and we had a rather custom rig for indexing content we were easily able to modify it to generate some JSON files in a format that Solr understands. The only tweak we had to make is flatten our data structure since Solr didn't supported nested elements like IDOL. All in all, not a big loss and in retrospect this kind of simplified things. However, the fact that it wasn't supported was kind of odd at first. Lets examine how both products index content and some of my favorite things about each one.

In IDOL world, we index content through the DIH that distributes the data, then each content engine indexes the data and it sits in the queue until the content engine performs a DRESYNC operation to commit it to the index. The search performance drag of DRESYNC was unacceptable in regular production operation. It would take forever and searches would slow down to a crawl. However, since we had a mirrored set-up, we could easily mark one of the content servers offline at the DAH (where queries come from), DRESYNC it while it’s counterpart responds to searches, then bring it up and perform the same operation on the other content engine. Other maintenance activities like DRECOMPACT were executed in a similar manner, naturally it was all automated and didn’t give us much trouble.

IDOL is generally used index unstructured content; binary documents such as PDF, MS-Word, Excel, HTML, etc... Through processing it will extract all document level metadata and content and index it in IDOL. Additionally, indexing is a distributed process, the server does not do ALL the heavy lifting. Generally, a repository specific connector (there is about 400 of them) will pick up the file and send it to Content Framework Server (CFS). CFS will process the document and run things like, text extraction, pre/post processing tasks, custom Lua scripts to massage the data and etc... Once everything is complete, it will send the data to IDOL for final indexing. As soon as IDOL receives the file, client's (CFSs) job is done and it is freed back into the wild for additional index tasks, even if the file is not indexed yet. IDOL then would index the content, but not commit it to the index until DRESYNC is ran.

With Solr, we were pleasantly surprised since the indexing and committing operations did not drag down the search performance, indexing was also a lot faster than IDOL. We used a very aggressive soft and hard commit policy that would commit all content within a few seconds of indexing. With Solr we no longer required this maintenance policy and can index content throughout the day without noticeably impacting performance. This allowed us to process large quantities of updates and changes during the day if there was a large backlog. 

The way solr commits the data to the index is pretty cool. In a nutshell, it will open a new segment of the index and write data to it without impacting the existing index. The frequency of writing indexed data from memory to disk is controlled by the hard commit interval. The frequency of committing data to the search index and making it searchable is a soft commit.  We were able to get optimal performance with 15 second hard commits and 5 minute soft commits. Once the soft commit is triggered, Solr will do whatever it needs to do with the data and open a new Searcher process that will be able to search the entire index, old segments and the new data that was indexed. Since the data indexing is not directly modifying the searcher process, search requests are not impacted by indexing. 

Where IDOL Wins:
When a Solr client POSTs files to the server, the indexing begins immediately, while the client connection is still open. This is not something I particularly like and I think this is one of the things that IDOL got right, with IDOL you are able to send all the data to the server and let it process the data when convenient. This frees up the client for additional work.

Additionally, IDOL allows you to configure multiple connector and CFS instances to distribute the indexing load across many systems. This is critical for some of the larger implementations where you are dealing with several terabytes of data, and indexing all data can take weeks or months.

Another win for IDOL is the amount of custom development required to index data. With IDOL, you can deploy, install and start indexing without any custom development, right out of the box. With Solr, you will need to write something to format the data in specific JSON format and send it to Solr for indexing. Additionally, Apache Tika provides a text extraction library that can extract text from binary formats and include it with your content. After this, you only need to develop something that crawls a repository or a filesystem location for new/changed files and indexes it into Solr.

Where Solr Wins:
Solr scores a few points in this category. First of all, indexing was a lot faster than IDOL, indexing all of our data took approximately 6 hours in IDOL, while with Solr we were able to cut that time down to about 4 hours.

Second, the index disk sizes were a lot smaller, somewhere in the range of 60% of what IDOL had to use.

Third, and I will mention this more later, Solr provides a lot of flexibility with data processing through different tokenizers and analyzers.

As always, I try to not play favorites here, I truly believe each product does what it does extremely well and depending on your specific requirements, one may work better than the other. Stay tuned, in the next post I will explore what each product allows you to do with the data once it is indexed.