A few days ago, suddenly in one of the SharePoint farms which my team maintains, after running fine for few months, it started logging the error message “SharePoint Crawl Error- Processing this item failed because of a timeout when parsing its contents” for almost all the content the crawler was trying to index.
It was a medium size farm, with 2 front end servers, 2 application servers with 2 database servers configured with always on availability groups. The search crawl was schedule to with “Continuous Crawl” with “Incremental Crawl” every 4 hours. Initially we thought it might be a temporary issue and would get resolved on it’s own during subsequent crawls, when the crawler runs during off-peak hours. We waited for a few days, but that never happened!
We started exploring a bit, first thing we found that whenever the crawl runs, in both the application servers the CPU utilization remained at about 20%-30%, but memory utilization reached 99%!
Further, most of the memory utilization was by processes called “noderunner.exe”. Some googling suggested the following changes in the noderunner.exe config file –
Go to C:\Program Files\Microsoft Office Servers\15.0\Search\Runtime\1.0\
Update <nodeRunnerSettings memoryLimitMegabytes=”0″ /> to something like <nodeRunnerSettings memoryLimitMegabytes=”250″ />. This is the configuration to limit NodeRunner process memory usage.
Well, personally I didn’t like the idea and did’t want to play with any configuration file which comes packaged with SharePoint, so I didn’t go for that.
Little bit more googling and found this old blog post in technet. It was meant for Sp 2013 RTM and we were dealing with SP 2016 with feature pack 1, so thought not so relevant. Anyway, half of the solution mentioned in this article was same which I had already decided not to move ahead with.
But, other half was interesting, it talked about changing Indexer performance setting in SharePoint using Set-SPEnterpriseSearchService PowerShell command. Now, that sounded interesting, so I explored that further and looked into the related technet article.
A quick read of the Performance Level parameter revealed that it specifies the relative number of threads for the crawl component performance.
The type must be one of the following values: Reduced, PartlyReduced, or Maximum. The default value is Maximum.
Maximum: Total number of threads = 32 times the number of processors, Max Threads/host = 8 plus the number of processors. Threads are assigned Normal priority.
Partly Reduced: Total number of threads = 16 times the number of processors , Max Threads/host = 8 plus the number of processors. Threads are assigned Below Normal priority.
Reduced: Total number of threads = number of processors, Max Threads/host = number of processors.
So, I went ahead and checked the current configuration in the SharePoint farm and as expected, it was found to be configured as Maximum.
Obviously, more the number of threads, more number of noderunner.exe instances resulting in more memory usage.
So, easy fix, just change it to PartlyReduced first and see if that solves the issue, so I went ahead and ran the PowerShell command in the SharePoint Management Shell –
Set-SPEnterpriseSearchService -PerformanceLevel PartlyReduced Restart-Service SPSearchHostController
Confirm that the Performance Level configuration has changed by running the following command
This solved the problem in our SharePoint environment. If you still get the error in a few files, you can further reduce the number of threads by using
Set-SPEnterpriseSearchService -PerformanceLevel Reduced Restart-Service SPSearchHostController
You don’t need to update the noderunner.exe config file manually. Less numbers of threads will take care of the memory utilization.
As a final touch, I increased the timeout value under Farm Search Administration to 600 from 60
And that’s it. All good now, all items getting crawled successfully and memory utilization in control too… happy days 🙂