How to handle ElasticSearch bulk queue full with requests from Jahia?
I have a Jahia feature like Augmented Search or jExperience that makes bulk requests to ElasticSearch. I noticed several errors in jCustomer logs like
2020-10-01T10:56:58,024 | ERROR | qtp2104759246-134433 | ElasticSearchPersistenceServiceImpl | 173 - org.apache.unomi.persistence-elasticsearch-core - 1.3.3.jahia | Error while executing in class loader org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of org.elasticsearch.transport.TransportService$7@8402600 on EsThreadPoolExecutor[bulk, queue capacity = 400, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@579979e2[Running, pool size = 2, active threads = 2, queued tasks = 424, completed tasks = 12664764]] at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:50) ~[173:org.apache.unomi.persistence-elasticsearch-core:1.3.3.jahia]
How can I adjust the queue size of my ElasticSearch to fix these errors?
You need to adjust your Thread pools settings in ElasticSearch for bulk operations.
For ElasticSearch please refer to this documentation.
For latest ElasticSearch versions please refer to this documentation.
One example for ElasticSearch 5.6 would be to change the following settings in file <ES_PATH>/conf/elasticsearch.yml :
thread_pool.bulk.queue_size: 800 thread_pool.bulk.size: 30
You need to restart ElasticSearch to make the changes effective.
Depending on the number of processors defined to your ElasticSearch cluster (default is 32) you might get the following error while starting ElasticSearch after the changes above:
2020-10-15T08:59:17,665][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler]  uncaught exception in thread [main] org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: Failed to parse value  for setting [thread_pool.bulk.size] must be <= 17
You should then adjust your thread_pool settings in accord to the number mentioned in the log: [thread_pool.bulk.size] must be <= 17