maxUrls

About this parameter

Limits the number of URLs your crawler can process.

Useful for demoing or preventing infinite link holes in the website structure.

maxUrls does not guarantee consistent indexing between crawls. The crawler processes in parallel, so discovered URLs can be processed in different orders for different crawls.

Examples

1
2
3
{
  maxUrls: 30,
}