Which analyzer does Jahia really use to index my content
Question
My site is not in English and full-text search behaves inconsistently. Searching chomage does not return content that writes "chômage". Searching assurance does not return content that writes "l'assurance". The same search returns a different number of results depending on the language selected in the JCR Query Tool.
repository.xml declares org.jahia.services.search.analyzer.EnglishSnowballAnalyzer. I changed it there and in both workspace.xml files, ran a full reindex, and nothing changed.
Which analyzer is actually used, and how do I change it?
Answer
The analyzer declared in the XML files is not the one used for full text
Three files declare an analyzer:
<TOMCAT>/webapps/ROOT/WEB-INF/etc/repository/jackrabbit/repository.xml <DATA>/repository/workspaces/default/workspace.xml <DATA>/repository/workspaces/live/workspace.xml
As soon as an indexing configuration is loaded, Jahia reads the default language of the platform at startup, looks up the matching entry in its analyzer registry, and replaces the full-text analyzer in memory. The files are never rewritten, so they keep displaying whatever you put there. Editing them does not change full-text behaviour, and running a full reindex afterwards changes nothing either.
Their value is not useless. It remains the analyzer of the fields that are not full text, and it is used as declared when no indexing configuration is loaded at all.
Older documentation mentions org.jahia.services.search.analyzer.DefaultLanguageAnalyzer. That class was removed in Jahia 8.0.0.0 and the upgrade patch replaces it with EnglishSnowballAnalyzer, which is why a standard installation now shows an English analyzer even for a non-English site.
The analyzer is selected by the default language of the platform
The selection is driven by one property only:
org.jahia.multilang.default_language_code
Its product default is en. It is a platform property. The j:defaultLanguage and j:languages properties of the site have no influence on it, so a site declared entirely in French can perfectly well have an index built in English, with nothing in the site administration to suggest it.
The registry is pre-populated for 13 languages without any configuration: ar, br, cjk, cn, cz, de, el, en, fa, fr, nl, ru, th. For instance fr maps to JahiaFrenchAnalyzer and en to an English Snowball analyzer. Accent folding is applied to full-text fields.
On Docker and Kubernetes installations, do not read this value in jahia.properties: an environment variable of the form jahia_cfg_* overrides the file. The effective value is the one displayed in Tools, System info (/modules/tools/systemInfo.jsp).
Indexing does not use a single analyzer
The analyzer is chosen document by document:
- translation nodes, which carry the internationalized properties, are indexed with the registry entry matching their own language;
- everything else, meaning non-internationalized properties, node names and the text extracted from files, is indexed with the analyzer of the platform default language.
This is what usually produces a half-working search. On a French site left with the default en, French content can be indexed correctly while non-internationalized properties and file content are indexed in English, and a French query never matches the second half. Aligning the platform language with the language of the content makes the whole index homogeneous, which is the real reason the property matters.
Queries
A full-text query is analyzed with the registry entry matching the language of the query. In the JCR Query Tool this is the language selector at the top of the page, and it defaults to English, which is a frequent source of misleading comparisons.
Starting with Jahia 8.2.3.0, the query text is analyzed twice, once with the language analyzer and once with the index analyzer, and the two are combined with an OR. A query in a registry language is therefore a superset. Up to 8.2.2.x only the language analyzer is used, so a mismatch between the analyzer that built the index and the analyzer that parses the query is much more visible. Do not transpose a search measurement from one of these versions to the other without keeping this in mind.
How to check which analyzer is actually running
Neither the configuration files nor the logs tell you: Jahia logs nothing about the analyzer it retains. Two ways to find out.
Run this script in Tools, Groovy console. It prints the analyzer in service and submits four discriminating words to it:
import org.apache.lucene.analysis.tokenattributes.CharTermAttribute
import org.apache.jackrabbit.core.query.lucene.FieldNames
import org.apache.jackrabbit.core.query.lucene.SearchIndex
import org.jahia.services.content.JCRSessionFactory
import org.jahia.settings.SettingsBean
def out = new StringBuilder()
def repo = JCRSessionFactory.getInstance().getDefaultProvider().getRepository().getRepository()
def c = repo.getClass(); def getWsInfo = null
while (c != null && getWsInfo == null) {
try { getWsInfo = c.getDeclaredMethod("getWorkspaceInfo", String.class) } catch (e) { c = c.getSuperclass() }
}
getWsInfo.setAccessible(true)
def tokens = { analyzer, text ->
def ts = analyzer.tokenStream(FieldNames.FULLTEXT, new java.io.StringReader(text))
def attr = ts.addAttribute(CharTermAttribute.class); ts.reset()
def l = []; while (ts.incrementToken()) l << attr.toString(); l.join(' ')
}
out << "Platform default locale: " << SettingsBean.getInstance().getDefaultLocale() << "\n"
["default", "live"].each { ws ->
def wsInfo = getWsInfo.invoke(repo, ws)
def getSm = wsInfo.getClass().getDeclaredMethod("getSearchManager"); getSm.setAccessible(true)
def index = getSm.invoke(wsInfo).getQueryHandler()
def fld = SearchIndex.class.getDeclaredField("analyzer"); fld.setAccessible(true)
def jackrabbitAnalyzer = fld.get(index)
def defFld = jackrabbitAnalyzer.getClass().getDeclaredField("defaultAnalyzer"); defFld.setAccessible(true)
def actual = defFld.get(jackrabbitAnalyzer)
def name = actual.getClass().getName()
try { def w = actual.getClass().getDeclaredField("wrappee"); w.setAccessible(true)
name = w.get(actual).getClass().getName() + " (accent folding applied)" } catch (e) { }
out << "\nWorkspace " << ws << "\n analyzer in use: " << name << "\n"
["chomage", "chômage", "assurance", "l'assurance"].each {
out << " '" << it << "' -> [" << tokens(index.getTextAnalyzer(), it) << "]\n"
}
}
return out.toString()
Reference outputs on a standard 8.2 installation, with no custom registry:
Platform default locale: en analyzer in use: org.apache.lucene.analysis.snowball.SnowballAnalyzer (accent folding applied) 'chomage' -> [chomag] 'chômage' -> [chomage] <- two distinct index terms 'assurance' -> [assur] 'l'assurance' -> [l'assur] <- they never meet Platform default locale: fr analyzer in use: org.jahia.services.search.analyzer.JahiaFrenchAnalyzer (accent folding applied) 'chomage' -> [chomag] 'chômage' -> [chomag] 'assurance' -> [asuranc] 'l'assurance' -> [asuranc]
In both cases repository.xml still declared EnglishSnowballAnalyzer.
If you cannot use the Groovy console, two searches answer the same question. On a content that writes "chômage", compare the number of results for chomage and for chômage: identical means a language analyzer is in use, different means an English one. Then search assurance on a content that writes "l'assurance": returned means elision is handled, absent means it is not.
How to change it
Set the property to the language of your content, for example:
org.jahia.multilang.default_language_code = fr
Optionally, declare the analyzer you want for that language at the end of your indexing_configuration.xml:
<analyzer-registry>
<fr class="org.jahia.services.search.analyzer.FrenchSnowballAnalyzer" useASCIIFoldingFilter="true"/>
</analyzer-registry>
Then run a full reindex. Partial or subtree reindexing does not rebuild the index with a new analyzer, and in a cluster each node holds its own local index, so every node must be reindexed. Use Tools, Search Engine Management (/modules/tools/search.jsp), or delete the index directories and restart:
<DATA>/repository/index <DATA>/repository/workspaces/default/index <DATA>/repository/workspaces/live/index
Keep in mind that the property is a platform property. It is also the fallback locale for JSTL formatting, the fallback used when a localized node is missing, and the default language when entering the site and after logout. On a monolingual platform all these uses go in the same direction, but validate the change outside production first.
Pitfalls
A single invalid line makes Jahia ignore the whole indexing_configuration.xml, and the only signal is a warning at startup:
WARN [SearchIndex] - Exception initializing indexing configuration from: .../indexing_configuration.xml javax.jcr.NamespaceException: Unknown namespace prefix nt. WARN [SearchIndex] - .../indexing_configuration.xml ignored.
Your <analyzer-registry> is then inactive, the analyzer of the XML files is really in service, and any measurement you make is about that analyzer. Check with grep -a "indexing_configuration.xml ignored" jahia.log* before concluding anything.
An analyzer class that cannot be loaded discards the entire registry, not just its own entry:
WARN [JahiaIndexingConfigurationImpl] - Couldn't process AnalyzerRegistry configuration
Full-text indexing then silently falls back to the analyzer declared in the XML files. This happens typically when a custom analyzer jar is removed from WEB-INF/lib while the configuration file still names its class. Always update the file first, check the analyzer in service, and remove the jar afterwards.
Check where your indexing_configuration.xml really lives. The path is given by the jahia.jackrabbit.searchIndex.workspace.config property. If that property is not set, Jahia loads its own standard file from the repository home, so a customized file placed elsewhere is simply never read, without any error.
A custom analyzer class cannot be delivered as an OSGi module. It is instantiated with Class.forName, from the web application class loader, so the jar has to be in WEB-INF/lib. Before going down that road, check whether one of the analyzers shipped with Jahia does the job: EnglishSnowballAnalyzer, FrenchSnowballAnalyzer, GermanSnowballAnalyzer, JahiaFrenchAnalyzer and StandardAnalyzer, in the org.jahia.services.search.analyzer package.
Choosing between two analyzers for the same language
Analyzers of the same language are not equivalent, and the difference is only visible on real words. The two French analyzers shipped with Jahia both handle elision but differ on accents and on how strongly they reduce words:
compared words FrenchSnowballAnalyzer JahiaFrenchAnalyzer salarié / salarie salari / salar DIFFERENT sala / sala same indemnité / indemnite indemn / indemnit DIFFERENT indemnit same chômage / chomage chomag same chomag same l'assurance / assurance assur same asuranc same employé / emploi emploi CONFLATED employ / emploi distinct référence / référent referent CONFLATED referenc/referent distinct cotisation cotis cot
FrenchSnowballAnalyzer applies accent folding after stemming, so it is only partially accent insensitive. JahiaFrenchAnalyzer uses a light stemmer that handles accents itself, so it is fully accent insensitive but conflates fewer word families while reducing words more aggressively. Decide on your own vocabulary, and decide before your test campaign: every analyzer change requires a full reindex.
Expect to lose a few results when moving to a language analyzer
Switching from an English index to a French one is an improvement overall, but some searches that used to work will stop working, and that is normal. An English analyzer applied to French text leaves most French suffixes untouched, which sometimes brings unrelated forms onto the same index term by accident:
Réglementation réglementaire Règlement règlements régler
EnglishSnowball reglement reglementair reglement reglement regler
FrenchSnowball reglement reglementair regl regl regl
JahiaFrench reglement reglementair regl regl regl
Searching "règlement" used to return content about "réglementation", because the English stemmer does not know the French -ement suffix and its -ation rule happened to produce the same term. A French stemmer strips -ement on one side and -ation on the other, so the two words no longer meet. Both French analyzers behave identically here: changing analyzer does not restore that match. Jackrabbit does ship a synonym provider (synonymProviderClass and synonymProviderConfigPath on SearchIndex), but synonym expansion only applies to a term explicitly prefixed with ~ in the query, which the Jahia search service never emits, so it is only reachable from your own queries.