Recently I came across a customer whose SharePoint 2007 crawl times were going through the roof. Incremental crawls used to take just some minutes but now took several days. It’s a fairly decent document corpus of about 2.5 million documents.
After checking the more obvious things, like the Crawl Log, I went to see if any performance tuning efforts were made that could impact the general performance of the indexing operation. There are two things to check here:
Setting the indexer performance level basically configures the number of threads available for crawling. Usually, the right choice when you have a dedicated index server (like my customer did), is setting it to “Maximum”.
Crawler Impact Rules can be used to reduce the impact of the crawl process on your external resources to prevent them from being overloaded with requests. By default, the number of documents that will be requested to a particular resource is set to 8 per request. By using a Crawler Impact rule you can reduce or increase this number. Usually (Noticed how I already used this word twice in this post?), increasing this number is not a good idea. My customer had a crawler impact rule to increase this to 32 documents per request - which I deleted to start my troubleshooting process.
To get some objective numbers on what was going on, I turned to my trusted performance monitor. These are the counters I used:
In my particular case, this all looked fairly decent. There is an excellent reference on TechNet on which counters to monitor, how to interpret them and some very good best practices for search performance in general - I strongly recommend to check it out! It tells you exactly what to look for.
Another tip is to use PAL to monitor SharePoint performance. You can use it to export a counters file that you can import into perfmon and what’s even better: you can let it analyze afterwards. It includes thresholds and tells you exactly where the problem may be. This tool should be on every SharePoint admin’s tool belt for sure! I will be detailing the use of PAL with SharePoint in a later post. What’s even better is that is has a performance counter set for SharePoint (2007) as well as SharePoint index servers specifically.
To get another insight into the problem, I increased the indexing logging level from Medium to Verbose. Be sure not to do this for more than a few minutes, because it can put a tremendous pressure on your performance.
Doing this provided me with some very useful information. In ULSViewer (never leave without it) I noticed a lot of entries like this:
mssdmn.exe - EnumerateListFolder is about to call WS for folder, strWebUrl …
mssdmn.exe - EnumerateListFolder received WS response, …
mssdmn.exe - InitListItem is about to call WS for item, strWebUrl …
mssdmn.exe - InitListItem received WS response, …
Each response event included a massive amount of XML data containing metadata (permissions and other properties) about the items being crawled - a folder or a list item from a document library in this example. This meant that although I could not see any activity in the crawl log, the indexing process was still going on - retrieving information on the changed items, folders or lists and processing them to the search database and index files.
So how many of these changes is he trying to process? How does he know what to process? What is the backlog of my indexing process? Figuring this out involves fiddling around with the SQL native tools - but that’s for my next post.