hostnameAliases

About this parameter

Defines mappings to replace given hostname(s).

Sometimes certain URLs are accessible from multiple domains, for example, due to a dynamic environment change (like staging vs. production) or hosting services like Netlify. You can use this parameter to define key-value pairs to dynamically replace one hostname with another when a site is being crawled. The key is the hostname to replace, and value is the hostname to replace it with.

Keys are exact match only. The Algolia Crawler doesn’t use micromatch for this parameter unlike others such as discoveryPatterns.

We apply the replacement whenever we encounter a URL with a matching hostname, whether through links in a sitemap or page, canonical URL, or redirect. For example, if you create a mapping for { "dev.algolia.com": "algolia.com" } and the crawler encounters https://dev.algolia.com/solutions/voice-search/, the crawler transforms this URL to https://algolia.com/solutions/voice-search/. If appropriate, we extract records and continue crawling from only the transformed URL, not the original version.

The mapping doesn’t apply on URLs listed in the startUrls, siteMaps, pathsToMatch or other configuration variables.

We also do not apply the mapping on the content extracted from a page. For example, if you extract text that includes the hostname to be replaced, we extract the original text without making a replacement.

Examples

1
2
3
4
5
{
  hostnameAliases: {
    'dev.algolia.com': 'algolia.com'
  }
}