300,000 pages possible with Netlify-Gatsby?

Is it possible to build and deploy a site with a very large number of pages? I need 300,000 pages. The pages are simple, just text, no graphics, a minimal UI.

(I’ve been using Netlify for over a year now, and I’ve had only great experience.) Thank you.

Wow! That’s a lot of pages! :sweat_smile:

Standard free plan Netlify builds cap out at 15 minutes total, any longer than that and the build won’t complete. If your Gatsby project can crank out all 300k in less than 15 minutes, then I don’t see any reason why Netlify won’t work for you. If you need more throughput in order to complete the build faster (or bandwidth to mitigate traffic to those 300,000 pages!!!) you might be in need of one of Netlify’s enterprise offerings.

I’m sure one of Netlify’s finest can pop in and add their thoughts, but in the meantime I’d maybe run some local build tests, maybe attempt a test build on Netlify, and if things don’t seem to be running smoothly, you can check out the price schedule here: https://www.netlify.com/pricing/

Best of luck!

1 Like

Some guru told you not to use a database?

It may be possible, as @noelforte said!

We can increase the build timeout limit from 15 minutes, so long as a card/billing details have been added to your team (as we would expect you to be more likely to exceed the limitations of the Starter plan’s inclusive build minutes). However, you’d want to see if you can sneak a build in under this first, of course!

Make sure you explore any applicable build plugins (netlify-plugin-gatsby-cache, for example) and assess whether they’re going to serve you well, too!

Hi Talal,

I’m on Netlify’s Business Development team and wanted to chime in…this is absolutely possible with Netlify! Between build optimizations, plug-ins, and tweaks we’ve worked with teams to successfully build and deploy similar large sites.

Feel free to reach out here and we can chat through the project and share how others are implementing best practices.
-Shafali

Not really. I’m not using a database, and I know no Gurus.

Great Suggestions. Thank you very much.

Thank you Shafali. Very good. I’m pursuing this project, and would love to gain from your experience. Let me share with you some more information:

  • Currently, I have a similar project, with 6300 pages. It builds and deploys on Netlify in about 30 minutes. I noticed, most of that time is not for build, but for post-build. And build-time is considerably shorter on my desktop PC.

  • Surely I do not have an enterprise account.

  • I contemplated building locally, then uploading the files to Netlify. I’m not sure this is a good practical approach.

Thank you very much. I look forward to hearing what you have to say.

great - we have reached out to you. If you have any other technical questions, come on back any time.

what SSG are you using? Hugo is supposedly the undisputed speed champion. However, I’m fond of Eleventy, and its performance is formidable.

1 Like

Good to know. Thank you. I’m using Gatsby.

We started with Gatsby too, but abandoned it for more flexible, less opinionated SSGs are which easier to use.

Maybe useful, this guy writes a bit about switching from Hugo to Gatsby to Eleventy.

Hello Netlify friends. I thought I’ll update you on my project with 300,000 pages.

As I expected, I encountered multiple roadblocks, and I overcame them. Yet eventually, Gatsby memory allocation problem. I researched it, I realized I won’t be able to overcome this. So I split the site into multiple builds, and wrote a script to loop through these builds, and I placed each build output in a subfolder.

While doing the above, I was thinking about the next challenge: deploy. Unfortunately, Netlify CLI deploy gave up, and gave me a JSON looking error message. I googled it, and realized I won’t be able to overcome it.

I could have uploaded the smaller sites independently, yet this has its own complications.

So I thought I’ll try Firebase hosting. Setting up was simple. The huge site uploaded smooth as cream, and blazing fast.

Surely, for all my upcoming not so huge projects, I’ll use Netlify.

I thought some of you can learn from my experience, so I’m posting this.

Thank you for the great Netlify product, and support. Have a good day.

1 Like

Thanks for that summary, @titani! Netlify is like a swiss army knife - useful for many, but not all, workflows, and this is useful information for folks with such large sites.

For folks who hit this thread in the future and wonder if they can fit their site on Netlify:

You’ll have to be very careful about large deploys, specifically:

  1. disabling as much asset optimization as possible on our side. (no article link for this one, but it will take “some time per file” which will be quite a lot of time, in case you have hundreds of thousands!)
  2. using appropriate resources for the build environment (cf [Support Guide] Making sure your builds use appropriate resources for Netlify's build system). Note that there is a (much) larger build environment available, in case that is needed for your site: https://www.netlify.com/pricing (look for “high performance builds”)
  3. …such as using appropriate plugins for gatsby (image handling ones are very unoptimized and unlikely to work here if you have very large images, for instance) (cf https://www.netlify.com/blog/2020/06/11/5-optimizations-for-faster-gatsby-builds/)
  4. make sure you aren’t changing every file with each build (cf [Support Guide] Making the most of Netlify's CDN cache)