Server responds different HTML content for different user-agent with curl

answered
#1

Hi guys,

I have been using gatsby + netlify so far, all is good, but now scratching my head to troubleshoot the problem with Facebook Share Debugger (the scraper).

If I do simple curl like this
curl https://www.zuyetawarmatrip.com/trips/2017-oct-pokhara-nepal/en/
It will return a valid HTML with the tag <!doctype html> at the beginning.

But for Facebook Share Debugger, to mimic the facebook external hit agent to scrape the page, I do this:
curl -v -A “facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)” “https://www.zuyetawarmatrip.com/trips/2017-oct-pokhara-nepal/en/

And what happens is instead of <!doctype html> tag, it echos the plain text “html” in the beginning, which creates an invalid HTML and defeats all Facebook meta OG tags I put.

So what may cause the problem? I doubt it should come from Netlify server side, not from Gatsby side (the output gatsby html are correct)

#3

Hi @zuyetawarmatik, that looks like a prerender bug. The current workaround is to remove your . Also, you can check out our post about debugging prerendering issues. Let me know how it goes.

#4

Great thanks, I have managed to solve it. Indeed my gatsby site content has full SSR rendering so I don’t think I need prerendering.

1 Like