IDOL To Solr Migration Lessons Learned - Part 5 Performance Tests Results

In this post, I will cover some lessons learned and key differences in performance of the two systems. Throughout the project, we’ve literally ran hundreds of tests to gauge stability and performance of both solutions under various conditions they would live in and the results were not always clear cut. Overall, it is hard to say which product performs better, but after some tweaking, we were able to get Solr to perform as good or better, in some cases, than IDOL; but we had to compromise on some functionality. 

First, let’s loop back to the requirements and cover architectures of both solutions. We have two servers, each one holds 4 content engines and a DAH. The content engines on the second server are a mirror copy of the data on the first server. This was replicated in SolrCloud, we deployed 4 shards with two replicas for each one. We also had another DAH that pointed to the two lower DAHs for queries in a round-robin fashion, so basically, all queries got load balanced across a set of four content engines on either box. This provided good performance as well as high availability. One point to re-iterate was that we were using IDOL 7, which is a few years behind. With SolrCloud, we didn’t have DAHs or DIHs, queries were went from the client to any of the replicas and they sorted out the result set and returned it back to the client. 

Now, let’s lay out our testing scenarios. We had a list of somewhere around 2,000 most frequent user queries that we used to hammer the systems. Our front end also supports multiple types of searches and each search can generate a few additional calls to Solr, all these were included in tests and although each call was separate, the times for execution were combined for related requests to get a bigger picture. Each test consisted of several load levels, most tests started at around 10,000 queries an hour and increased by 5,000 or 10,000 each 30 minutes. The maximum capacity we tested was approximately 60,000 queries per hour. 

Since our performance goal was to do as good as IDOL, we started with defining IDOL performance by running a few tests. IDOL performance was stellar for each individual query, however because in IDOL facet requests are a separate call, we had to combine the time for both requests. This added some overhead, but combined they were still was less than half a second.

While IDOL tests were running, we were monitoring query performance as well as system utilization. The system utilization of RAM and CPU was essentially linear, each content engine was given about 10GB of RAM for caching and internal workings and it never surpassed that. The CPU utilization on these 128 core boxes was almost always below 25%, even at peak load. 

As we pushed the system beyond our initial design requirements, we noticed a sudden increase in query lag. This was due to our specified thread count for each DAH, the threads were maxed out and as such searches had to wait a little while to be executed. Naturally, if this was a production environment we would deploy a few more DAHs to support increase in search, but since it wasn’t our testing objective and we were short on time, we decided to just call it a day. The main element here is that the IDOL content engines weren’t really breaking a sweat throughout most of these tests. 

We repeated all of these tests with Solr, with necessary adjustment for internal workings on the system. However, it is worth to note that all business rules, fields, weights and etc… were preserved and replicated in Solr as close we as could match them. 

As we begun to test Solr, with our limited experience in the system, we immediately started to run into problems. The initial and most critical problem was system stability; Solr kept crashing during our tests. The culprit ended up being Java Garbage Collection pauses which caused SolrCloud to time out from ZooKeeper. Our handy consultants were able to help us address this issue, however the documentation on this kind of problem is minimal at best. We would never have been able to track this as a root cause on our own. 

If you are running into stability issues with Solr, Garbage Collection is definitely worth investigating. One of confirming symptoms is that there is some significant pauses in the log file with no activity whatsoever followed by a time out from zookeeper and a few funny error messages. 

Once the stability issues were addressed we had to examine query performance. We were getting some very mixed results on some queries and it took us a little while to determine the exact cause. We were seeing generally good performance, better than IDOL, but on some queries the system would just stall out and take between 3 and 5 seconds to respond. 

This is where we had to sacrifice some functionality. We have a unique identifier that is shared between several documents; when searching, we generally only want to see the data related to the unique identifier, however data from individual items is searchable as well. In both systems, this kind of grouping it is a piece of cake since we can easily group by a field value, however the culprit was getting facet counts based on this grouping. For some of the broader terms, the result sets returned were pretty large, somewhere around a million and Solr had to calculate facet counts based on groups which caused this delay. We had to get customer’s approval to turn off this functionality and were able to improve Solr’s performance at a slight loss in functionality. 

We also measured system performance during Solr’s tests. I think it was worse than IDOL’s, the CPU utilization was never stable and resembled what I would imagine to be a crack head’s EKG chart.  JVM heap was configured at 32GB as well, totaling less than IDOL. I wish we could give it some more RAM, but due to garbage collection problems we were advised against it. 

In the end, we were able to shave off about ¼ of a second for almost all of our searches on average, which is a pretty tangible improvement in my opinion. However, we never went back to test IDOL’s performance without counting up facets based on groups, so I can’t say of certain which one is faster.

So here is my take on it. As far as performance goes, I think Solr is faster than IDOL 7, mainly because you can retrieve facets and results in the same query. However, as far as performance bang for the buck, I think IDOL does a much better job here. It’s hardware resource utilization is much more controlled and predictable. 

Additionally, I my personal belief is that SolrCloud automates too much which turns into a lack of flexibility and control. For example, in IDOL, we had direct control over which content servers will be processing requests. While in SolrCloud, requests for each shard are distributed among replicas in an unpredictable manner. Don’t get me wrong, I am all in favor of automation, but the control freak in me is screaming “NOOOOOOOO!!!!”. 

I know that there are a lot of elements missing here since query performance and system utilization is heavily dependent on data and types of queries and I didn’t share that information with you yet, but don’t worry, more info will be coming soon.  

IDOL To Solr Migration Lessons Learned - Part 2 Indexing Data

Since the data we were indexing into IDOL was plain old XML and we had a rather custom rig for indexing content we were easily able to modify it to generate some JSON files in a format that Solr understands. The only tweak we had to make is flatten our data structure since Solr didn't supported nested elements like IDOL. All in all, not a big loss and in retrospect this kind of simplified things. However, the fact that it wasn't supported was kind of odd at first. Lets examine how both products index content and some of my favorite things about each one.

In IDOL world, we index content through the DIH that distributes the data, then each content engine indexes the data and it sits in the queue until the content engine performs a DRESYNC operation to commit it to the index. The search performance drag of DRESYNC was unacceptable in regular production operation. It would take forever and searches would slow down to a crawl. However, since we had a mirrored set-up, we could easily mark one of the content servers offline at the DAH (where queries come from), DRESYNC it while it’s counterpart responds to searches, then bring it up and perform the same operation on the other content engine. Other maintenance activities like DRECOMPACT were executed in a similar manner, naturally it was all automated and didn’t give us much trouble.

IDOL is generally used index unstructured content; binary documents such as PDF, MS-Word, Excel, HTML, etc... Through processing it will extract all document level metadata and content and index it in IDOL. Additionally, indexing is a distributed process, the server does not do ALL the heavy lifting. Generally, a repository specific connector (there is about 400 of them) will pick up the file and send it to Content Framework Server (CFS). CFS will process the document and run things like, text extraction, pre/post processing tasks, custom Lua scripts to massage the data and etc... Once everything is complete, it will send the data to IDOL for final indexing. As soon as IDOL receives the file, client's (CFSs) job is done and it is freed back into the wild for additional index tasks, even if the file is not indexed yet. IDOL then would index the content, but not commit it to the index until DRESYNC is ran.

With Solr, we were pleasantly surprised since the indexing and committing operations did not drag down the search performance, indexing was also a lot faster than IDOL. We used a very aggressive soft and hard commit policy that would commit all content within a few seconds of indexing. With Solr we no longer required this maintenance policy and can index content throughout the day without noticeably impacting performance. This allowed us to process large quantities of updates and changes during the day if there was a large backlog. 

The way solr commits the data to the index is pretty cool. In a nutshell, it will open a new segment of the index and write data to it without impacting the existing index. The frequency of writing indexed data from memory to disk is controlled by the hard commit interval. The frequency of committing data to the search index and making it searchable is a soft commit.  We were able to get optimal performance with 15 second hard commits and 5 minute soft commits. Once the soft commit is triggered, Solr will do whatever it needs to do with the data and open a new Searcher process that will be able to search the entire index, old segments and the new data that was indexed. Since the data indexing is not directly modifying the searcher process, search requests are not impacted by indexing. 

Where IDOL Wins:
When a Solr client POSTs files to the server, the indexing begins immediately, while the client connection is still open. This is not something I particularly like and I think this is one of the things that IDOL got right, with IDOL you are able to send all the data to the server and let it process the data when convenient. This frees up the client for additional work.

Additionally, IDOL allows you to configure multiple connector and CFS instances to distribute the indexing load across many systems. This is critical for some of the larger implementations where you are dealing with several terabytes of data, and indexing all data can take weeks or months.

Another win for IDOL is the amount of custom development required to index data. With IDOL, you can deploy, install and start indexing without any custom development, right out of the box. With Solr, you will need to write something to format the data in specific JSON format and send it to Solr for indexing. Additionally, Apache Tika provides a text extraction library that can extract text from binary formats and include it with your content. After this, you only need to develop something that crawls a repository or a filesystem location for new/changed files and indexes it into Solr.

Where Solr Wins:
Solr scores a few points in this category. First of all, indexing was a lot faster than IDOL, indexing all of our data took approximately 6 hours in IDOL, while with Solr we were able to cut that time down to about 4 hours.

Second, the index disk sizes were a lot smaller, somewhere in the range of 60% of what IDOL had to use.

Third, and I will mention this more later, Solr provides a lot of flexibility with data processing through different tokenizers and analyzers.

As always, I try to not play favorites here, I truly believe each product does what it does extremely well and depending on your specific requirements, one may work better than the other. Stay tuned, in the next post I will explore what each product allows you to do with the data once it is indexed.