Tools
/
Crawler
/
ignoreRobotsTxtRules
Jan. 08, 2021
ignoreRobotsTxtRules
Type:
boolean
Default:
false
Parameter syntax
ignoreRobotsTxtRules: true|false
About this parameter
When set to true, the crawler will ignore rules set in your robots.txt.
Examples
Copy
1
2
3
{
ignoreRobotsTxtRules: true,
}
