Solr - configuration for Chinese and correct results for german umlauts

Revision as of 09:26, 13 October 2014

configuration for Chinese

This is a configuration of the schema.xml (/WEB-INF/solr/conf/)for the OpenCms Solr search for a Chinese website

<analyzer type="index">
     <tokenizer class="solr.StandardTokenizerFactory"/>
     <filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
     <tokenizer class="solr.StandardTokenizerFactory"/>
     <filter class="solr.LowerCaseFilterFactory"/>
     <filter class="solr.PositionFilterFactory" />
</analyzer>

Configuration for Chinese search with Solr

<analyzer type="index">
   <tokenizer class="solr.StandardTokenizerFactory"/>
   <filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
    <tokenizer class="solr.StandardTokenizerFactory"/>
    <filter class="solr.LowerCaseFilterFactory"/>
    <filter class="solr.PositionFilterFactory" />
</analyzer>

Alternative configuration

<analyzer>
   <tokenizer class="solr.SmartChineseSentenceTokenizerFactory"/>
   <filter class="solr.SmartChineseWordTokenFilterFactory"/>
   <filter class="solr.LowerCaseFilterFactory"/>
   <filter class="solr.PositionFilterFactory" />
</analyzer>

This configuration didn’t work for us, because the indexing in the Admin area of OpenCms didn’t even start with this config. But it was a suggestion from someone on the OpenCms mailing list.

Additional info

The [Solr wiki] says, that the PositionFilterFactory should only be used at query time.

Further configuration (in Tomcat)

Our main problem was, not finding the correct indexer. It was the proper configuration of Tomcat. As the solr search sends get requests, we had to add the correct URI encoding in the server.xml:

   <Connector connectionTimeout="20000" port="8080" protocol="HTTP/1.1" redirectPort="8443" URIEncoding="UTF-8"/>

On our production sever we use the AJP connector, so we had to add it there as well:

   <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" URIEncoding="UTF-8"/>

This setting solved the problem when searching for words with german umlauts (e.g. müller) as well. As we got search results with the wrong settings, it was hard to spot, that this were the wrong results.