IDOL To Solr Migration Lessons Learned - Part 1 Requirements

Recently my organization completed a full IDOL 7 to SOLR migration for an e-commerce application and I wanted to share some of the lessons learned with you as well as my take on Solr and IDOL. If you’ve been reading my blog, you know that I am very keen on requirements, so let’s start with that.

• Document Count: Around 40 million
• Churn: Anywhere between 10,000 and 200,000 a night, must be capable of indexing/changing a million products over a weekend.
• Performance: 75% of searches in IDOL were under half a second, next 20% were under ¾ of a second, the rest took a bit longer. Generally, we see anywhere between 200,000 and 300,000 searches per day. They peak between 9am and 4pm, at which point, we see between 20,000 and 30,000 an hour, with a historical maximum of around 46,000 . Solr was to perform better than IDOL.
• Relatively heavy use of grouping, faceting and sorting.
• Hardware: Two servers (for HA), each one with 128 cores and 315 GB of RAM. We also had dedicated storage for each collection/idol content engine.

Naturally, there were other requirements, but for the purpose of this series we will stick with this.