Sitecore 重建索引错误，未知字段 'title_t_vi' 'xxx_t_vi' , Sitecore rebuild index error unknown field 'title_t_vi' 'xxx_t_vi'

由system 发表于 2022-03-16 | 更新于 2022-03-25 | 文章分类:Sitecore | 463 次阅读 | 0 条评论

引用

https://sitecoreblog.marklowe.ch/2018/10/customize-solr-managed-schema/
https://mikael.com/2020/10/dealing-with-solr-managed-schema-through-sitecore-config-files/
https://github.com/mikaelnet/sitecore-solr-config
https://sitecore.namics.com/adding-new-languages-to-sitecores-solr-indexes/
https://solr.apache.org/guide/6_6/language-analysis.html

错误信息 error message

Job started: Index_Update_IndexName=sitecore_master_index|#Exception: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> SolrNet.Exceptions.SolrConnectionException: <?xml version="1.0" encoding="UTF-8"?>
<response>

<lst name="responseHeader">
  <int name="status">400</int>
  <int name="QTime">23</int>
</lst>
<lst name="error">
  <lst name="metadata">
    <str name="error-class">org.apache.solr.common.SolrException</str>
    <str name="root-error-class">org.apache.solr.common.SolrException</str>
  </lst>
  <str name="msg">ERROR: [doc=sitecore://master/{eb4abdf4-67f5-45bd-b9fd-1ed3dbbd2959}?lang=vi-vn&amp;ver=1&amp;ndx=sitecore_master_index] unknown field 'title_t_vi'</str>
  <int name="code">400</int>
</lst>
</response>
 ---> System.Net.WebException: The remote server returned an error: (400) Bad Request.
   at System.Net.HttpWebRequest.GetResponse()
   at HttpWebAdapters.Adapters.HttpWebRequestAdapter.GetResponse()
   at SolrNet.Impl.SolrConnection.GetResponse(IHttpWebRequest request)
   at SolrNet.Impl.SolrConnection.PostStream(String relativeUrl, String contentType, Stream content, IEnumerable`1 parameters)
   --- End of inner exception stack trace ---
   at SolrNet.Impl.SolrConnection.PostStream(String relativeUrl, String contentType, Stream content, IEnumerable`1 parameters)
   at SolrNet.Impl.SolrConnection.Post(String relativeUrl, String s)
   at SolrNet.Impl.LowLevelSolrServer.SendAndParseHeader(ISolrCommand cmd)
   at Sitecore.ContentSearch.SolrProvider.SolrBatchUpdateContext.Commit()
   at Sitecore.ContentSearch.SolrProvider.SolrSearchIndex.PerformRebuild(Boolean resetIndex, Boolean optimizeOnComplete, IndexingOptions indexingOptions, CancellationToken cancellationToken)
   at Sitecore.ContentSearch.SolrProvider.SolrSearchIndex.Rebuild(Boolean resetIndex, Boolean optimizeOnComplete)
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at Sitecore.Reflection.ReflectionUtil.InvokeMethod(MethodInfo method, Object[] parameters, Object obj)
   at Sitecore.Jobs.JobRunner.RunMethod(JobArgs args)
   at (Object , Object )
   at Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args)
   at Sitecore.Pipelines.DefaultCorePipelineManager.Run(String pipelineName, PipelineArgs args, String pipelineDomain, Boolean failIfNotExists)
   at Sitecore.Pipelines.DefaultCorePipelineManager.Run(String pipelineName, PipelineArgs args, String pipelineDomain)
   at Sitecore.Jobs.DefaultJob.DoExecute()
   at Sitecore.Abstractions.BaseJob.ThreadEntry(Object state)

环境 environments

  • Sitecore 10.1.0
  • Solr 8.4.0

重现问题步骤

  • 在sitecore(/sitecore/system/Languages) 中添加越南语言(vi-VN)
  • 给越南语言添加内容，并保存发布，
  • 尝试重建sitecore_master_index索引

screenshots



问题分析

出现这个问题是因为solr默认不支持越南语言分词(vi-VN), 在这个页面里面有solr默认支持语言分词列表: https://solr.apache.org/guide/6_6/language-analysis.html, 但是没有越南语言
我们在solr的配置文件中也可以看到语言相关的配置，打开具体索引的managed-schema(C:\xxx(solr根目录)\server\solr\sitecore_master_index\conf)
但是并没有 fieldType text_vi, dynamicField *_t_vi, 所以我要在managed-schema中添加上越南语言的配置

...
<fieldType name="text_lv" class="solr.TextField" positionIncrementGap="100">
    <analyzer>
      <tokenizer class="solr.StandardTokenizerFactory"/>
      <filter class="solr.LowerCaseFilterFactory"/>
      <filter class="solr.StopFilterFactory" words="lang/stopwords_lv.txt" ignoreCase="true"/>
      <filter class="solr.LatvianStemFilterFactory"/>
    </analyzer>
  </fieldType>
  <fieldType name="text_nl" class="solr.TextField" positionIncrementGap="100">
    <analyzer>
      <tokenizer class="solr.StandardTokenizerFactory"/>
      <filter class="solr.LowerCaseFilterFactory"/>
      <filter class="solr.StopFilterFactory" format="snowball" words="lang/stopwords_nl.txt" ignoreCase="true"/>
      <filter class="solr.StemmerOverrideFilterFactory" dictionary="lang/stemdict_nl.txt" ignoreCase="false"/>
      <filter class="solr.SnowballPorterFilterFactory" language="Dutch"/>
    </analyzer>
  </fieldType>
  <fieldType name="text_no" class="solr.TextField" positionIncrementGap="100">
    <analyzer>
      <tokenizer class="solr.StandardTokenizerFactory"/>
      <filter class="solr.LowerCaseFilterFactory"/>
      <filter class="solr.StopFilterFactory" format="snowball" words="lang/stopwords_no.txt" ignoreCase="true"/>
      <filter class="solr.SnowballPorterFilterFactory" language="Norwegian"/>
    </analyzer>
  </fieldType>
....
  <dynamicField name="*_t_bg" type="text_bg" indexed="true" stored="true"/>
  <dynamicField name="*_t_ca" type="text_ca" indexed="true" stored="true"/>
  <dynamicField name="*_t_cs" type="text_cz" indexed="true" stored="true"/>
  <dynamicField name="*_t_da" type="text_da" indexed="true" stored="true"/>
  <dynamicField name="*_t_de" type="text_de" indexed="true" stored="true"/>
  <dynamicField name="*_t_el" type="text_el" indexed="true" stored="true"/>
  <dynamicField name="*_t_es" type="text_es" indexed="true" stored="true"/>
...

尝试解决方式一

越南语言相关配置，注：由于我也不懂越南语言，只是从网络中查找，并不能准确的对越南语言进行分词，只保证可能正常重建索引

  <fieldType name="text_vi" class="solr.TextField" positionIncrementGap="100">
    <analyzer>
      <tokenizer class="solr.StandardTokenizerFactory"/>
      <filter class="solr.LowerCaseFilterFactory"/>
      <filter class="solr.WordDelimiterFilterFactory" preserveOriginal="0"/>
    </analyzer>
  </fieldType>

  <dynamicField name="*_t_vi" type="text_vi" indexed="true" stored="true"/>

直接把上面的配置文件添加到 sitecore_master_index 索引中的 managed-schema 配置文件中
重启Solr, 重启solr后，手动添加的配置才会有效果
尝试重建master索引成功，看似解决了问题。

在Populate Solr Managed Schema master索引之后 ，再次重建索引又失败了，提示相同的问题，检查发现，我们手动添加的越南配置文件在populate solr managed schema之后 dynamicField <dynamicField name="*_t_vi" type="text_vi" indexed="true" stored="true"/>丢失了，

尝试解决方式二

从方式一可以看出，手动添加配置，可以工作， 但是在populate solr managed schema之后，配置文件会丢失，应该managed schema重新自动生成了，但是因为没有vi-VN的配置， 所以重新生成之后并不有vi-VN的配置，
是否可以添加sitecore pipeline，在populate solr managed schema时添加上vi-VN的配置呢？在经过搜索后面发现是可以的
参考链接（上面已经贴出来了）：
https://sitecoreblog.marklowe.ch/2018/10/customize-solr-managed-schema/
https://github.com/mikaelnet/sitecore-solr-config
由于sitecore 10 和9相关代码还是有一些有区别的，所以我结合上面两种，修改了可用于sitecore 10的相关代码。

  1. fieldTypes config
<fieldTypes>
    <add-or-replace-field-type>
        <name>text_vi</name>
        <class>solr.TextField</class>
        <positionIncrementGap>100</positionIncrementGap>
        <analyzer>
            <tokenizer>
                <class>solr.StandardTokenizerFactory</class>
            </tokenizer>
            <filters>
                <class>solr.LowerCaseFilterFactory</class>
            </filters>
            <filters>
                <class>solr.WordDelimiterFilterFactory</class>
                <preserveOriginal>0</preserveOriginal>
            </filters>
        </analyzer>
    </add-or-replace-field-type>
</fieldTypes>
  1. dynamicField config
<fields>
    <add-dynamic-field>
        <name>*_t_vi</name>
        <type>text_vi</type>
        <indexed>true</indexed>
        <stored>true</stored>
    </add-dynamic-field>
</fields>
  1. Sitecore pipeline config
<pipelines>
    <contentSearch.PopulateSolrSchema>
        <!-- Processor for additional SOLR schema -->
        <processor type="Feature.Search.Pipelines.CustomSchemaPopulator, Feature.Search"
                   patch:after="processor[@type='Sitecore.ContentSearch.SolrProvider.Pipelines.PopulateSolrSchema.PopulateFields, Sitecore.ContentSearch.SolrProvider']">
            <param type="Feature.Search.Pipelines.CustomSchemaPopulateHelperFactory" />
        </processor>
    </contentSearch.PopulateSolrSchema>
</pipelines>

完整的配置文件

<?xml version="1.0"?>

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
    <sitecore>
        <contentSearch>
            <customSolrManagedSchema>
                <commands applyToIndex="not-to-use">
                    <fieldTypes>
                        <add-or-replace-field-type>
                            <name>text_vi</name>
                            <class>solr.TextField</class>
                            <positionIncrementGap>100</positionIncrementGap>
                            <analyzer>
                                <tokenizer>
                                    <class>solr.StandardTokenizerFactory</class>
                                </tokenizer>
                                <filters>
                                    <class>solr.LowerCaseFilterFactory</class>
                                </filters>
                                <filters>
                                    <class>solr.WordDelimiterFilterFactory</class>
                                    <preserveOriginal>0</preserveOriginal>
                                </filters>
                            </analyzer>
                        </add-or-replace-field-type>
                    </fieldTypes>
                    <fields>
                        <add-dynamic-field>
                            <name>*_t_vi</name>
                            <type>text_vi</type>
                            <indexed>true</indexed>
                            <stored>true</stored>
                        </add-dynamic-field>
                    </fields>
                </commands>
            </customSolrManagedSchema>
        </contentSearch>
        <pipelines>
            <contentSearch.PopulateSolrSchema>
                <!-- Processor for additional SOLR schema -->
                <processor type="Feature.Search.Pipelines.CustomSchemaPopulator, Feature.Search"
                           patch:after="processor[@type='Sitecore.ContentSearch.SolrProvider.Pipelines.PopulateSolrSchema.PopulateFields, Sitecore.ContentSearch.SolrProvider']">
                    <param type="Feature.Search.Pipelines.CustomSchemaPopulateHelperFactory" />
                </processor>
            </contentSearch.PopulateSolrSchema>
        </pipelines>
    </sitecore>
</configuration>

完整的代码

public class CustomSchemaPopulator : PopulateFields
    {
        public CustomSchemaPopulator(IPopulateHelperFactory populateFactory) : base(populateFactory)
        {
        }
    }

    public class CustomSchemaPopulatorHelper : ISchemaPopulateHelper
    {
        private readonly SolrSchema _solrSchema;

        public CustomSchemaPopulatorHelper(SolrSchema solrSchema)
        {
            _solrSchema = solrSchema;
        }

        public IEnumerable<XElement> GetAllFields()
        {
            var elements = new List<XElement>();

            foreach (XmlNode commands in Factory.GetConfigNodes("contentSearch/customSolrManagedSchema/commands/fields"))
            {
                elements.AddRange(commands.ChildNodes.Cast<XmlNode>().Select(n => ParseCommand(n.OuterXml, _solrSchema)));
            }

            return elements;
        }

        private XElement ParseCommand(string xml, SolrSchema schema)
        {
            var element = XElement.Parse(xml);

            // set "add" or "replace" depending if field type already exists in schema
            if (element.Name == "add-or-replace-field-type")
            {
                var fieldTypeName = element.Elements().FirstOrDefault(e => e.Name == "name");
                if (fieldTypeName != null && !string.IsNullOrEmpty(fieldTypeName.Value))
                {
                    if (schema.SolrFieldTypes.Any(f => f.Name == fieldTypeName.Value))
                    {
                        element = new XElement("replace-field-type", element.Attributes(), element.Elements());
                    }
                    else
                    {
                        element = new XElement("add-field-type", element.Attributes(), element.Elements());
                    }
                }
            }

            return element;
        }

        public IEnumerable<XElement> GetAllFieldTypes()
        {
            var elements = new List<XElement>();

            foreach (XmlNode commands in Factory.GetConfigNodes("contentSearch/customSolrManagedSchema/commands/fieldTypes"))
            {
                elements.AddRange(commands.ChildNodes.Cast<XmlNode>().Select(n => ParseCommand(n.OuterXml, _solrSchema)));
            }

            return elements;
        }
    }

    public class CustomSchemaPopulateHelperFactory : IPopulateHelperFactory
    {
        public ISchemaPopulateHelper GetPopulateHelper(SolrSchema solrSchema)
        {
            return new CustomSchemaPopulatorHelper(solrSchema);
        }
    }

总结

从最后的配置文件和代码来看，也很容易理解，就是在sitecore populate solr managed schema时添加上自定义的一些配置文件，
我们在配置文件里添加了一个fieldType text_vi 和一个dynamicField *_t_vi
然后在代码里读取相应的配置节点并把它们添加到sitecore.

发表评论

关闭
  • Special Field must have a value

评论
Reply

还没有人评论，抢个沙发吧...

Viagle Blog

欢迎来到我的个人博客网站

Categories

Linux Book Android v2ray Sitecore Python Mathematic Matlab Tomcat Raspberry Pi Shadowsocks Database GIT

友情链接

Github 廖雪峰 钱奇杰 XX妹子

Powered by Awesome Python Blog. Copyright © .

www.viagle.cn. All rights reserved.