Configuration

The ID of the application you want to store the crawler extractions in.

API key for your targeted application.

Prefix added to the names of all indices defined in the crawler’s configuration.

Number of concurrent tasks per second that can run for this configuration.

How often a complete crawl should be performed.

The crawler uses these URLs as entry points to start crawling.

URLs found in sitemaps are treated as startUrls for the crawler: they are used as starting points for the crawl.

When set to true, the crawler will ignore rules set in your robots.txt.

Whether the Crawler should extract records from a page whose robots meta tag contains noindex or none.

Whether the Crawler should follow links marked as nofollow (i.e., with the rel=”nofollow” tag) and extract links from a page whose robots meta tag contains nofollow or none.

Whether the Crawler should extract records from a page that has a canonical URL specified.

URLs found in extraUrls are treated as startUrls for your crawler: they are used as starting points for the crawl.

Limits the processing of URLs to the specified depth, inclusively.

Limits the number of URLs your crawler can process.

Whether to save a backup of your production index before it is overwritten by the index generated during a crawl.

When true, all web pages are rendered with a chrome headless browser. The crawler will use the rendered HTML.

Defines the settings for the indices that the crawler updates.

Tells the crawler which URLs to ignore or exclude.

Filters out specified query parameters from crawled URLs. This can help you avoid indexing duplicate URLs.

Modify all crawler’s requests behavior.

Override the default logic used to extract URLs from pages.

Defines external data sources you want to retrieve during every crawl and make available to your extractor function.

This property defines how the crawler acquires a session cookie.

A configurable collection of safety checks to make sure the crawl was successful.

Determines which web pages are translated into Algolia records and in what way.

Indicates additional web pages that the Crawler should visit.

Defines mappings to replace given hostname(s).

Turn crawler’s cache on or off.