IDOL To Solr Migration Lessons Learned - Part 5 Performance Tests Results

In this post, I will cover some lessons learned and key differences in performance of the two systems. Throughout the project, we’ve literally ran hundreds of tests to gauge stability and performance of both solutions under various conditions they would live in and the results were not always clear cut. Overall, it is hard to say which product performs better, but after some tweaking, we were able to get Solr to perform as good or better, in some cases, than IDOL; but we had to compromise on some functionality. 

First, let’s loop back to the requirements and cover architectures of both solutions. We have two servers, each one holds 4 content engines and a DAH. The content engines on the second server are a mirror copy of the data on the first server. This was replicated in SolrCloud, we deployed 4 shards with two replicas for each one. We also had another DAH that pointed to the two lower DAHs for queries in a round-robin fashion, so basically, all queries got load balanced across a set of four content engines on either box. This provided good performance as well as high availability. One point to re-iterate was that we were using IDOL 7, which is a few years behind. With SolrCloud, we didn’t have DAHs or DIHs, queries were went from the client to any of the replicas and they sorted out the result set and returned it back to the client. 

Now, let’s lay out our testing scenarios. We had a list of somewhere around 2,000 most frequent user queries that we used to hammer the systems. Our front end also supports multiple types of searches and each search can generate a few additional calls to Solr, all these were included in tests and although each call was separate, the times for execution were combined for related requests to get a bigger picture. Each test consisted of several load levels, most tests started at around 10,000 queries an hour and increased by 5,000 or 10,000 each 30 minutes. The maximum capacity we tested was approximately 60,000 queries per hour. 

Since our performance goal was to do as good as IDOL, we started with defining IDOL performance by running a few tests. IDOL performance was stellar for each individual query, however because in IDOL facet requests are a separate call, we had to combine the time for both requests. This added some overhead, but combined they were still was less than half a second.

While IDOL tests were running, we were monitoring query performance as well as system utilization. The system utilization of RAM and CPU was essentially linear, each content engine was given about 10GB of RAM for caching and internal workings and it never surpassed that. The CPU utilization on these 128 core boxes was almost always below 25%, even at peak load. 

As we pushed the system beyond our initial design requirements, we noticed a sudden increase in query lag. This was due to our specified thread count for each DAH, the threads were maxed out and as such searches had to wait a little while to be executed. Naturally, if this was a production environment we would deploy a few more DAHs to support increase in search, but since it wasn’t our testing objective and we were short on time, we decided to just call it a day. The main element here is that the IDOL content engines weren’t really breaking a sweat throughout most of these tests. 

We repeated all of these tests with Solr, with necessary adjustment for internal workings on the system. However, it is worth to note that all business rules, fields, weights and etc… were preserved and replicated in Solr as close we as could match them. 

As we begun to test Solr, with our limited experience in the system, we immediately started to run into problems. The initial and most critical problem was system stability; Solr kept crashing during our tests. The culprit ended up being Java Garbage Collection pauses which caused SolrCloud to time out from ZooKeeper. Our handy consultants were able to help us address this issue, however the documentation on this kind of problem is minimal at best. We would never have been able to track this as a root cause on our own. 

If you are running into stability issues with Solr, Garbage Collection is definitely worth investigating. One of confirming symptoms is that there is some significant pauses in the log file with no activity whatsoever followed by a time out from zookeeper and a few funny error messages. 

Once the stability issues were addressed we had to examine query performance. We were getting some very mixed results on some queries and it took us a little while to determine the exact cause. We were seeing generally good performance, better than IDOL, but on some queries the system would just stall out and take between 3 and 5 seconds to respond. 

This is where we had to sacrifice some functionality. We have a unique identifier that is shared between several documents; when searching, we generally only want to see the data related to the unique identifier, however data from individual items is searchable as well. In both systems, this kind of grouping it is a piece of cake since we can easily group by a field value, however the culprit was getting facet counts based on this grouping. For some of the broader terms, the result sets returned were pretty large, somewhere around a million and Solr had to calculate facet counts based on groups which caused this delay. We had to get customer’s approval to turn off this functionality and were able to improve Solr’s performance at a slight loss in functionality. 

We also measured system performance during Solr’s tests. I think it was worse than IDOL’s, the CPU utilization was never stable and resembled what I would imagine to be a crack head’s EKG chart.  JVM heap was configured at 32GB as well, totaling less than IDOL. I wish we could give it some more RAM, but due to garbage collection problems we were advised against it. 

In the end, we were able to shave off about ¼ of a second for almost all of our searches on average, which is a pretty tangible improvement in my opinion. However, we never went back to test IDOL’s performance without counting up facets based on groups, so I can’t say of certain which one is faster.

So here is my take on it. As far as performance goes, I think Solr is faster than IDOL 7, mainly because you can retrieve facets and results in the same query. However, as far as performance bang for the buck, I think IDOL does a much better job here. It’s hardware resource utilization is much more controlled and predictable. 

Additionally, I my personal belief is that SolrCloud automates too much which turns into a lack of flexibility and control. For example, in IDOL, we had direct control over which content servers will be processing requests. While in SolrCloud, requests for each shard are distributed among replicas in an unpredictable manner. Don’t get me wrong, I am all in favor of automation, but the control freak in me is screaming “NOOOOOOOO!!!!”. 

I know that there are a lot of elements missing here since query performance and system utilization is heavily dependent on data and types of queries and I didn’t share that information with you yet, but don’t worry, more info will be coming soon.  

IDOL To Solr Migration Lessons Learned - Part 4 Stuff I miss about IDOL

There is going to be much more good info on Solr coming up, especially next week's post which will cover our process for relevance tuning and things we learned. But, in the mean time I wanted to take a stroll down memory lane and mention some of the things I miss about IDOL. These items, although not major, still come up to haunt me now as well as during the deployment. Oh, and I promise, I am not getting any money from HP for this. 

Let’s start off with an easy one; documentation. When you buy IDOL, you get a plethora of manuals, so much, in fact, that you could club a baby seal to death with them. There is a getting started guide, to ease you into IDOL, Server Admin documentation as well as much reference material that will provide you everything you need to know about every single type of call you can make to IDOL. It seems intimidating at first, but it is truly priceless. It is up to date, consistent and informative.

With Solr, we were pretty much at the mercy of online forums, some blog posts and the wiki pages. The wiki pages provided some good information, but some areas definitely need improvement. Some parameters only had a sentence or a few words of a description which drove me nuts. Forums support was good, but then again, you have to provide quite a bit of information about what you are trying to do, reasons for which may be hidden in some old business requirements that don't always make sense. Naturally, that information has to be disclosed because the first ten replies will echo a message of “Well, whachya doing that for?”. 

We procured support for Solr and it came in handy, we couldn't have done it without those guys. However, at times I found myself reaching out to them with a question for which there was a very valid answer that was not documented anywhere. A bit frustrating, but something workable.

Another thing is logging. In IDOL, there are several distinct logs that are written out, they are marvelous, and they tell me exactly what it is doing. There is one for queries, there is another one for the application, another for indexing, and a few others and they are all neat, informative and concise, plus I can tune logging detail per type of log.

With Solr, there are several loggin modules that can be configured to your liking and log level for different kinds of items is very adjustable, however I failed to find a good balance between the right information and log volume. Maybe, I need to spend a bit more time on it, but this process is much simpler in IDOL.

Aside from general log upkeep, there is another point of IDOL that I terribly miss. IDOL logs tell you exactly what is happening. With Solr, I found it to be much much more obscure. For example, if the core is performing some internal indexing operation, there is absolutely no status reporting, it’s the equivalent of kicking off an action and just waiting … log file equivalent of those old school Windows hourglass that just spun around and round. If indexing actions weren't bad enough, wait until a core goes down, recovery kicks in and the customer asks you how long it will take.

I also miss some of IDOL’s terms functionality, when I first started working with IDOL, I read the sales material and thought to myself.. “yea, right….”. Years of sweating (or shivering) in server rooms and having to deliver on sale’s people’s imagination made me a natural skeptic when it comes to these kinds of claims.

However, after a while, I was able to validate these claims of conceptual understanding myself. With IDOL, you are able to scan a piece of text for the best terms that represent it in comparison to other documents in the index. It sounds like TF-IDF, but it is much much more than that. I am not sure how it works, but it came in handy when I was trying to group similar types of content together.

While we are talking about terms, I think I should mention IDOL stemming which works better than Porter and Snowball. In my book, they either do too much or not enough. IDOL’s stemming is still the best I’ve seen. 

The last and very important point is that there are few relevance configurations that I miss.  First of all, in IDOL document scores in the results are relative. In Solr, they are not because of QueryNorm, in Solr you can't perform any kind of meaningful comparison between two sets of results. IDOL does this, with IDOL you can run multiple queries and compare the scores of the results with each other. Now, I know what you are thinking... Why would anyone want to do that? Well, when it comes down to grouping related content to finding conceptually similar stuff this is really important.

Sorry for the trip down memory lane, just couldn’t resist it. There are a few other items on my wishlist for Solr, I will cover them in the next posts. I hope you guys are getting some value from this, leave a comment or two to let me know what you think.

IDOL To Solr Migration Lessons Learned - Part 3 Data Processing

This post is a quick follow up to the last one in which I discussed several differences between IDOL and Solr when it comes to indexing data. In this post we will explore how different products maintain the data and what they allow you to do with it.

With IDOL were able to set certain fields for various searches, such as IndexFields for text indexing, or parametric for other types of searches. Any kind of data massaging had to happen before indexing to IDOL; via custom development or a Lua script in CFS. This certainly got the job done, but it didn’t even come close to the flexibility that Solr offers. With Solr, we customized processing for each field using Solr’s built in analyzers and tokenizes. This treats each field independently and tokenize (create individual searchable elements) by field. After tokenization, we run several analyzers on fields. Analyzers allow you to massage the data a bit before it is indexed; things like stemming, synonyms, minimum token length filtering occurs at this stem.

For example, we have a filed for manufacturer part number, the data comes in a variety of different formats and no single rule would accommodate all manufacturers and their favorite preference for part number formats. However, we were able to take it up a notch, but using the Word Delimiter Filter (Analyzer) to massage a bit each token in the manufacturer part number field. The word delimiter filter is configured to break up a token at transition from numeric to characters and vice versa. This breaks up patters such as FLV12345X, then, the length filter is applied to filter out anything less than two characters, so this field would be indexed as FLV12345X (original term is kept), FLV and 12345.  If a user searches for a part number as it is indexed, there will be a hit on three terms in one field, all of these terms should hit. Since the field is weighted highly, that almost guarantees that the product will be displayed. This came in handy, since we don’t have control over our data and get even the same part numbers in different formats from each vendor.

An even cool part, is that you can specify two sets of processes for each field, index and query level processing. This allows you to run a specific process on data when you are indexing it, and then a slightly different process when you are querying it. This came in handy for synonyms as we didn’t want to expand the query to contain synonyms, since some of them are abbreviations and they could dilute the search results, so instead we just included them at index time and indexed a few extra terms for each picked up synonym.

Solr’s list of tokenizes and analyzers can be found here: https://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters

Don’t worry if the list looks intimidating at first, Solr’s admin interface lets you test how various strings will look like with different field configurations. This allows you to easily test any field level processing without having to drum up some dummy content, index it, and then search for it until you figure out what is what. The Analyzer tab in Solr UI lets you see exactly what will happen with your data when you index it.

To sum up and keep it short: Solr is definitely a winner in this category. IDOL has some other cool functionality, especially when dealing with conceptual understanding of various terms, but it's not the same as this awesomeness of Solr field level processing.

I hope you found this post informative, I think that field processing is by far the coolest thing about Solr; but there are several others that are pretty close behind it. If you liked this post and want to get future posts by me, remember to sign up in the top right.