renderJavaScript

About this parameter

When true, all web pages are rendered with a chrome headless browser. The crawler will use the rendered HTML.

Because rendering JavaScript-based web pages is slower than crawling regular HTML pages, you can apply this setting to a specified list of micromatch URL patterns. These patterns can include negations and wildcards.

With this setting enabled, JavaScript is executed on the webpage by a Chrome headless browser. Because a lot of websites have infinite refreshes and updates, the browser has a timeout of 10 seconds: for now, this value cannot be modified .

Enabling this feature can lead to inconsistent records across crawls, depending on the browser load and the website speed.

Make sure your JavaScript-enabled pages load fast enough for the crawler.

Examples

1
2
3
{
  renderJavaScript: true,
}

1
2
3
4
5
{
  renderJavaScript: [
    'http://www.mysite.com/dynamic-pages/**',
  ],
}