Code a la Mode banner

How to Change Render Behavior During Crawl-Time (When Using Coveo)

In this post:
  1. 1. Why does crawler context matter
  2. 2. Let’s code!
    1. 2.1. Wrap the default HtmlContentInBodyWithRequestsProcessor
    2. 2.2. Configuration patch
    3. 2.3. Set up renderings

You know how it’s super easy to change how a rendering is rendered based on Sitecore.Context.PageMode? If you’ve ever wished there was a similar check available for PageMode.IsIndexCrawlerCrawling, then read on.
User VS Index Crawler

Why does crawler context matter

  1. If you have renderings that show alerts to front-end users (for example, cookie consent or legal disclaimers), it makes sense not to render these at crawl time since their HTML shouldn’t show up in search results.
  2. If you have dynamic renderings that AJAX their content, you may want to provide a static version for the index.
  3. If you have renderings that inject user tracking code, it makes sense to keep this code out of the indexed version of the page.

Let’s code!

Wrap the default HtmlContentInBodyWithRequestsProcessor

By default, Coveo uses the HtmlContentInBodyWithRequestsProcessor class to index a page’s content. This class makes an HTTP request to the page’s URI and indexes the HTML that is returned. Unfortunately, HtmlContentInBodyWithRequestsProcessor is mostly non-virtual and private methods, so it’s difficult to customize. But we can get around this by creating a wrapper.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
using System;
using System.IO;
using Coveo.Framework.Configuration;
using Coveo.Framework.Http;
using Coveo.Framework.Processor;
using Coveo.SearchProvider.Pipelines;
using Coveo.SearchProvider.Processors;

namespace Website.Search.Processors
{
/// <summary>
/// Wraps <see cref="HtmlContentInBodyWithRequestsProcessor"/> to add query string
/// parameter to Http Request that denotes the request as coming from index crawler.
/// </summary>
public class CustomHtmlContentInBodyWithRequestsProcessor
: IProcessor<CoveoPostItemProcessingPipelineArgs>
{
private readonly object _lock = new object();

private HtmlContentInBodyWithRequestsProcessor _baseProcessor;

/// <inheritdoc/>
public void Process(CoveoPostItemProcessingPipelineArgs args)
{
if (_baseProcessor == null)
{
lock (_lock)
{
if (_baseProcessor == null)
{
_baseProcessor = new HtmlContentInBodyWithRequestsProcessor(new CustomItemContentFetcher(new HttpRequestBuilder(), args.IndexConfiguration));
}
}
}

_baseProcessor.Process(args);
}

private class CustomItemContentFetcher : IItemContentFetcher
{
private readonly HtmlItemContentFetcher _baseFetcher;

public CustomItemContentFetcher(IHttpRequestBuilder requestBuilder, CoveoIndexConfiguration indexConfiguration)
{
_baseFetcher = new HtmlItemContentFetcher(requestBuilder, indexConfiguration);
}

public Stream FetchItemContent(string url) => _baseFetcher.FetchItemContent(FixUrl(url));

public IHttpWebResponse FetchHttpWebResponse(string url) => _baseFetcher.FetchHttpWebResponse(FixUrl(url));

/// <summary>
/// Adds a "isIndexCrawl" query string parameter to the item's default URI
/// </summary>
private string FixUrl(string url)
{
var uriBuilder = new UriBuilder(url);
var query = System.Web.HttpUtility.ParseQueryString(uriBuilder.Query);
query["isIndexCrawl"] = "1";
uriBuilder.Query = query.ToString();
return uriBuilder.ToString();
}
}
}
}

Configuration patch

Replace default processor with custom processor in configuration

1
2
3
4
5
6
7
8
9
10
11
12
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
<sitecore>
<pipelines>
<coveoPostItemProcessingPipeline>
<processor type="Coveo.SearchProvider.Processors.HtmlContentInBodyWithRequestsProcessor, Coveo.SearchProviderBase">
<patch:delete />
</processor>
<processor type="Website.Search.Processors.CustomHtmlContentInBodyWithRequestsProcessor, Website" />
</coveoPostItemProcessingPipeline>
</pipelines>
</sitecore>
</configuration>

Set up renderings

Now we just need to make our renderings aware of whether the special query string parameter is present. Add this IsCrawl property to your Rendering Model class. You can test for this property in your View and change rendering output accordingly.

1
public bool IsIndexCrawler => Sitecore.Context.Request.QueryString["isIndexCrawl"] == "1";

Bon Appétit!