[Common Issue] Understanding and debugging prerendering

unanswered
#1

With the advent of modern web hosting aka the JAMStack, some robots and crawlers need “help” navigating client-side links. This problem first presented itself initially when navigation was implemented using Javascript instead of HTML, but these days the OpenGraph standard and unfurling by social sharing services seems to be in many cases more important to our customers. (For more on this, see a further reading link we have added to the bottom of this post.) Prerendering is one solution to this problem.

Netlify provides a prerendering service described in some detail in this article. This post aims to help if it doesn’t “just work” and you’d like to debug or understand how to test.

Which requests are prerendered?

Prerendered content is only displayed in certain situations. Specifically it:

  • will only be used for a Netlify website once prerendering is enabled on your site’s build & deploy settings page
  • will only be returned in response to a web request when there is a User-Agent sent with the request that needs this special content, or a special URL is used (see below for more details around manual testing and the special URL).

There are a few dozen user agents that a request can present that will cause a prerendered response, for instance Twitterbot and facebookexternalhit/1.0 . All crawlers that we are aware of which need prerendering receive this treatment automatically, but if yours seems to be missing, please mention it below so we can potentially add it to the list!

Testing the service with your content

To test things out manually, you can also request a prerendered URL - see particularly the part about _escaped_fragment_ in our prerendering documentation.

So, to test prerendering, this is what I do. You can use any User-Agent that we prerender for, so I use twitterbot in this example.

curl -A twitterbot https://www.sitename.com/

or

curl https://www.sitename.com/?_escaped_fragment_=

This should show different output than not using a special User Agent or the ?_escaped_fragment_= style URL, which will demonstrate that the service is in effect.

Some caveats about the prerender service

  1. We cache the prerendered content for 24 to 48 hours, so you will want to make sure to test with different URLs each time if you are making changes and republishing a lot while you get things dialed in. Most importantly, this cache is NOT invalidated by a new deploy of your site!

  2. You may also need to use window.prerenderReady in your javascript to indicate to our service when it is safe to consider the page fully rendered to take the snapshot for the services that need it, as mentioned in the prerender.io docs here. If this isn’t set, we take a snapshot after 10 seconds regardless of the state of the page.

  3. When testing with Google Webmaster tools, please be aware that there can be an issue using the “Fetch as Google” functionality. This tool does not check for the fragment meta tag and re-request the page with the ?escaped_fragment= query parameter as it does in production: Googlebot itself does not have that issue. So, when using “Fetch as Google” you’ll need to append the ?_escaped_fragment_ query parameter on the end of the URL that you are testing, and that should show your prerendered page. If that works and if you have the following code snippet in the <head> of your HTML, then the real Googlebot will work just fine.

        <meta name="fragment" content="!">
    

Next Steps

That’s a lot of advice, but it’s possible that it still won’t resolve your issue. So what can you do next if your content doesn’t seem to be rendering correctly after enabling the prerender service? After checking out the below caveats to make sure it’s not related to one of those, here’s a bit more advice:

First, be aware that your code needs to provide the OpenGraph tags that are needed for social sharing services. A lot of frameworks and plugins (e.g. react-helmet) handle setting them for you, but “a wrong og tag” is generally not the fault of the prerender service but rather your code. You can examine the source code that we serve as described above to debug what is happening.

If instead you’re seeing errors (HTTP 5xx responses), then it is likely that your code is not rendering at all. Since we don’t expose prerender logs to customers, using a local copy of Netlify’s prerender code (forked from prerender.io, and different!) and following the local-use instructions in its readme is the best way to debug those issues in depth. This will show any syntax errors that are encountered as well as letting you examine the output including any OpenGraph tags.

A customer wrote up his journey in great detail with a lot of practical advice in this medium article here - it explains a lot about the world of rendering and unfurling and testing using third party tools. If you’re struggling to understand why prerendering is needed and how prerendering works with social sharing articles, this is a great starting place.

1 Like
Server responds different HTML content for different user-agent with curl
#3

This no longer works for the facebookexternalhit/1.1 useragent, or bing.
It currently works for Googlebot and the Twitterbot/1.0 useragent.

Please help bring this back.