The "more content" widget cannot be sorted to show the most recent entries

The more content widget which uses the solr "more like this" endpoint for related content does not seem to honour the sort parameter, we wanted to show the latest entries

so for example 

#set ($query = "%{ Sort = 'date desc', PageSize = $pageSize, PageIndex = $pageIndex}")
#if ($filters.Length > 0)
$query.Add('Filters', $filters)
#end

## perform query
#set ($items = false)
#set ($items = $core_v2_searchResult.GetRelatedContent($currentContent.ContentId.ToString(), $query))


I have tried this will all combinations of sort value

 I have a couple of solr logs to confirm its getting to solr, but doesn’t seem to effect the results ?           

2021-01-12 14:32:05.198 INFO  (qtp315932542-19) [   x:verint-content] o.a.s.c.S.Request [verint-content]  webapp=/solr path=/mlt params={mm=0&q=id:"c8656e93-356c-439b-9f2f-bf68cecd0d57"&fq=%2Btype:(blog)+%2Bgroup:(1)&fq=%2Broles:(1+OR+2+OR+3+OR+4+OR+8+OR+10+OR+11+OR+12+OR+16+OR+22+OR+25+OR+26+OR+29+OR+32+OR+36+OR+39+OR+43+OR+46+OR+49+OR+52+OR+55+OR+58+OR+61+OR+64+OR+67+OR+73+OR+833+OR+839+OR+848+OR+868+OR+869+OR+878+OR+891+OR+905+OR+911+OR+1007+OR+1018+OR+1029+OR+1044+OR+1050+OR+1089+OR+1106+OR+1165+OR+1171+OR+1242+OR+1290+OR+1338+OR+1343+OR+2343+OR+2347+OR+2350+OR+2353+OR+2379+OR+2462+OR+2463+OR+2464+OR+2480+OR+2497+OR+2500+OR+2503+OR+2504+OR+2505+OR+2507+OR+2529+OR+2547+OR+2553+OR+2559+OR+2583+OR+2625+OR+2626+OR+2627+OR+2628+OR+2629)

&sort=date+asc&rows=2} status=0 QTime=4


2021-01-12 14:32:24.314 INFO  (qtp315932542-14) [   x:verint-content] o.a.s.c.S.Request [verint-content]  webapp=/solr path=/mlt params={mm=0&q=id:"c8656e93-356c-439b-9f2f-bf68cecd0d57"&fq=%2Btype:(blog)+%2Bgroup:(1)&fq=%2Broles:(1+OR+2+OR+3+OR+4+OR+8+OR+10+OR+11+OR+12+OR+16+OR+22+OR+25+OR+26+OR+29+OR+32+OR+36+OR+39+OR+43+OR+46+OR+49+OR+52+OR+55+OR+58+OR+61+OR+64+OR+67+OR+73+OR+833+OR+839+OR+848+OR+868+OR+869+OR+878+OR+891+OR+905+OR+911+OR+1007+OR+1018+OR+1029+OR+1044+OR+1050+OR+1089+OR+1106+OR+1165+OR+1171+OR+1242+OR+1290+OR+1338+OR+1343+OR+2343+OR+2347+OR+2350+OR+2353+OR+2379+OR+2462+OR+2463+OR+2464+OR+2480+OR+2497+OR+2500+OR+2503+OR+2504+OR+2505+OR+2507+OR+2529+OR+2547+OR+2553+OR+2559+OR+2583+OR+2625+OR+2626+OR+2627+OR+2628+OR+2629)

&sort=date+desc&rows=2} status=0 QTime=2

 

I suspect that the solr /mlt end point does not support sorting after a bit of googling

Has anyone else tried this ?

Parents
  • Is the requirement strictly order by date?

    The mlt handler ([solr]/mlt) used by Community does not support sorting but the mlt query parser ({!mlt}id) does. That does not help you out much now but there are options.

    1. Filter the results to a specific time period - ex. the past 1-2 years. The filters work fine (they are already used for types) and should be good results if you are looking for more recent results though its still sorted by score.

    2. New search handler - This would involve creating a new solr handler via solr config file (ex. /mlt2) that uses the mlt query parser and intercepting the search call with an automation to use this handler. Additionally, you could still filter by date and layer a boost with large emphasis on recent content -- this layers relevancy with date without completely ignoring scoring. This version seems like a better option for the product as well. I could put together an example of a handler if you go this route.
  • Thanks Kevin, the trigger for this is that old content is being shown as a next read even though newer content has been added

    It's from Jerome, but the second option seems like its worth a try, if you can provide an example change we can try it locally or in stage and see how it performs

    My concern with filters is that we may get no results, but we could make a second more relaxed call to solr in those cases, so a 2 staged approach

  • To implement:

    1. Copy the handler below (mlt2) to solrconfig.xml. Look for the existing /mlt and put it around there for simplicity. Note: You do not need to stop the service.
    2. Goto SolrAdmin -> CoreAdmin -> telligent-content and 'reload' the core.
    3. Automation - create an automation that subscribes to the Solr.BeforeSearch event that looks for an mlt query and replaces it with mlt2. Attached in an example you can import of something that adjust boost but its not a solution for this just a start.
      1. string replace "/mlt" - "/mlt2"
      2. You MUST copy the document id value from q (string split id:<doc-value>) to additional query parameter of "qq". Reason is explained below. If you dont do this it wont work. So an example final value for "qq" would be "qq=71df682e-5393-437e-a6ef-1658a6041f8b".

    Other notes:

    • Variables - Anything you see in the "q" element below in the format of $${variable}} can be overridden in th querystring when sending to Solr. There are default values of course already set.
    • Date boosting - Date boosting is in the handler now but if you plan on sorting by date you can use the commented out "q" element. Alternatively if you want to date boost conditionally - ex. ignore on blogs, you could sniff the type filter and set "&dateBoost=1" which effectively ignores it.
    • The mapping of the existing query (in this case the document id) to 'qq' in the query string is for security. We are controlling the query by "stealing" the q parameter and using the default parser which allows incoming query parsers to be used which open ended is not desirable. This is common, good practice for controlled scenarios like this one.
    • We cannot guarantee the format of the url will stay the same across releases so keying of '/mlt' is fine but I cannot guarantee it would stay the same for years to come. No plans on changing but it could happen. This is also a plus for new handler because it would fallback to original.

     <requestHandler name="/mlt2" class="solr.SearchHandler">
        <lst name="defaults">
          <str name="echoParams">explicit</str>
          <str name="rows">10</str>
          <str name="df">title</str>
          <str name="mltFields">title</str>  <!-- content -->
          <str name="queryFields">title</str>
          <str name="boostA">10</str>
          <str name="boostB">.1</str>
          <str name="dateBoost">recip(ms(NOW/HOUR,date),3.16e-11,$${boostA},$${boostB})</str>
          <str name="qq"></str> <!-- set value here for testing -->
        </lst>
          <lst name="invariants">
              <str name="q">
                {!boost b=$${dateBoost}}{!mlt qf=$${queryFields} mintf=1 fl=$${mltFields} minwl=3 mindf=1 boost=true interestingTerms=details}$${qq}
              </str> 
              <!-- use this version if you do not want date boosting at all -->
              <!-- <str name="q">
                {!mlt qf=$${queryFields} mintf=1 fl=$${mltFields} minwl=3 mindf=1 boost=true interestingTerms=details}$${qq}
              </str>  -->
              <str name="wt">xml</str>
      </lst>
    

    <automation name="Site Search Adjuster" version="11.0.0.0" description="Changes boosting on site searches" id="9b0aa8b334854ef5a2f0000b590e7396" executeAsServiceUser="false" isSingleton="false" trigger="Event" lastModified="2021-01-13 20:19:07Z" provider="31f486cc-03b5-42b6-b5df-81cdd976aa59">
    	<events>
    		<event key="solr.beforesearch" />
    	</events>
    	<executionScript language="JavaScript"><![CDATA[var solrUrl = context_v2_automationTrigger.Arguments.SolrOptions.Url;
    context_v2_automationTrigger.Arguments.SolrOptions.Url += "&boost=" + core_v2_encoding.UrlEncode("query({! v='type:blog^=10'},1)");
    //core_v2_eventLog.Write(context_v2_automationTrigger.Arguments.SolrOptions.Url);]]></executionScript>
    </automation>

Reply
  • To implement:

    1. Copy the handler below (mlt2) to solrconfig.xml. Look for the existing /mlt and put it around there for simplicity. Note: You do not need to stop the service.
    2. Goto SolrAdmin -> CoreAdmin -> telligent-content and 'reload' the core.
    3. Automation - create an automation that subscribes to the Solr.BeforeSearch event that looks for an mlt query and replaces it with mlt2. Attached in an example you can import of something that adjust boost but its not a solution for this just a start.
      1. string replace "/mlt" - "/mlt2"
      2. You MUST copy the document id value from q (string split id:<doc-value>) to additional query parameter of "qq". Reason is explained below. If you dont do this it wont work. So an example final value for "qq" would be "qq=71df682e-5393-437e-a6ef-1658a6041f8b".

    Other notes:

    • Variables - Anything you see in the "q" element below in the format of $${variable}} can be overridden in th querystring when sending to Solr. There are default values of course already set.
    • Date boosting - Date boosting is in the handler now but if you plan on sorting by date you can use the commented out "q" element. Alternatively if you want to date boost conditionally - ex. ignore on blogs, you could sniff the type filter and set "&dateBoost=1" which effectively ignores it.
    • The mapping of the existing query (in this case the document id) to 'qq' in the query string is for security. We are controlling the query by "stealing" the q parameter and using the default parser which allows incoming query parsers to be used which open ended is not desirable. This is common, good practice for controlled scenarios like this one.
    • We cannot guarantee the format of the url will stay the same across releases so keying of '/mlt' is fine but I cannot guarantee it would stay the same for years to come. No plans on changing but it could happen. This is also a plus for new handler because it would fallback to original.

     <requestHandler name="/mlt2" class="solr.SearchHandler">
        <lst name="defaults">
          <str name="echoParams">explicit</str>
          <str name="rows">10</str>
          <str name="df">title</str>
          <str name="mltFields">title</str>  <!-- content -->
          <str name="queryFields">title</str>
          <str name="boostA">10</str>
          <str name="boostB">.1</str>
          <str name="dateBoost">recip(ms(NOW/HOUR,date),3.16e-11,$${boostA},$${boostB})</str>
          <str name="qq"></str> <!-- set value here for testing -->
        </lst>
          <lst name="invariants">
              <str name="q">
                {!boost b=$${dateBoost}}{!mlt qf=$${queryFields} mintf=1 fl=$${mltFields} minwl=3 mindf=1 boost=true interestingTerms=details}$${qq}
              </str> 
              <!-- use this version if you do not want date boosting at all -->
              <!-- <str name="q">
                {!mlt qf=$${queryFields} mintf=1 fl=$${mltFields} minwl=3 mindf=1 boost=true interestingTerms=details}$${qq}
              </str>  -->
              <str name="wt">xml</str>
      </lst>
    

    <automation name="Site Search Adjuster" version="11.0.0.0" description="Changes boosting on site searches" id="9b0aa8b334854ef5a2f0000b590e7396" executeAsServiceUser="false" isSingleton="false" trigger="Event" lastModified="2021-01-13 20:19:07Z" provider="31f486cc-03b5-42b6-b5df-81cdd976aa59">
    	<events>
    		<event key="solr.beforesearch" />
    	</events>
    	<executionScript language="JavaScript"><![CDATA[var solrUrl = context_v2_automationTrigger.Arguments.SolrOptions.Url;
    context_v2_automationTrigger.Arguments.SolrOptions.Url += "&boost=" + core_v2_encoding.UrlEncode("query({! v='type:blog^=10'},1)");
    //core_v2_eventLog.Write(context_v2_automationTrigger.Arguments.SolrOptions.Url);]]></executionScript>
    </automation>

Children
No Data