I wondered if I could ask for some help on understanding why my Gatsby builds are taking so long to deploy. From my deploy log I can see that:
build.command completed in 2m 38s
Netlify Build completed in 3m 41s
Finished processing build request in 13m10.280336458s
**Build time: 13m 48s. Total deploy time: 50m 38s**
I suspect there is an issue which I need to debug as I made a simple change to a single file which shouldn’t have affected anything else yet on my most recent build I get:
**15498 new files uploaded**
15146 generated pages and 352 assets changed.
I’ve tried to download the .zip of the deployed files for the last 2 builds but I get an error each time (could this be due to the size of my site).
We’re considering splitting the site into 2 and proxying one half but I’d still like to understand why the process takes far longer than my initial build time.
Thanks in advance for any help and please let me know if you need more detail.
Do you have a lot of assets that are being optimized? Or a massive form?
i can say that it is very likely that your site download is getting stuck due to the site size - that functionality works awesome for small sites, and is somewhat unreliable for large ones, which, hopefully, we will fix at some point, but that clearly isn’t your main issue - your main issue is site builds!
here is some maybe helpful info:
if you aren’t able to make any dents in the time it takes with that info, let us know and we’ll think on it some more.
Thanks - I’ll re-read those posts and see if I can see anything obvious.
Assets - most of the images are served off an external cdn - we have 15,000 pages and each has at least one image on them. Does any processing happen on these in your build process? No large forms.
What happens in the ‘Post processing HTML stage’ - looking at the build log this is where we have the hang:
4:16:47 PM: Finished processing build request in 13m10.280336458s
4:17:27 PM: Starting post processing
4:17:28 PM: Post processing - HTML
4:54:17 PM: Post processing - redirect rules
4:54:17 PM: Post processing - header rules
4:54:17 PM: Post processing done
4:54:17 PM: Site is live
I should add that asset optimization is disabled in the GUI for this build.
Is there anything else you guys can tell me that could help? Locally it builds in a reasonable time and as we can see from the logs the initial build is comparatively quick.
Hi, @tim-line, there is also forms processing. Does the site in question use our Forms feature?
If it doesn’t use Forms, it might be worth disabling the forms processing as well to see if this improves the post processing time. Our support team can make that settings change so please let us know if you would like us to do so.
If so, please tell us the site subdomain (subdomain-here.netlify.app) or the API ID from “Site Name” > Settings > General > Site information.
We have forms on the site but we’re not using Netlify forms so yes please. The API I’d is - 32e107b2-a262-4b8c-adcd-63b23b59565a.
I assume it must be something happening in the Netlify steps rather than the Gatsby build? The only other thought was we have some large XML files we process as part of the build but these are outside or the ‘src’ and ‘static’ folders so aren’t being compiled at runtime and assume wouldn’t be touched by Netlify?
Oh wow - this made an incredible difference. This isn’t a strict comparison as the later build will be building more pages but here’s the result.
Build with forms post processing on: Build time: 13m 48s. Total deploy time: 50m 38s
Build with forms post processing off: Build time: 18m 55s. Total deploy time: 18m 56s
This is a large site (> 15000 pages) so I’m sure daily builds shouldn’t even be this long.
I appreciate the help in resolving this but it’s a little frustrating to have this issue in the first place. Is there a reason this isn’t disabled by default or configurable with the other post processing options? Is there anything else you can do to speed up deploys?
Hmm it does seem like, if one file changed and you have the Gatsby cache plugin (which you do!), then we shouldn’t be rebuilding every page of your site. I wonder if there is some conflict between how that plugin works and how the caching works for this other plugin that you also seem to be using, the Onix Source Plugin:
10:58:12 PM: $ gatsby build
10:58:17 PM: success open and validate gatsby-configs - 0.085s
10:58:18 PM: success load plugins - 1.041s
10:58:18 PM: Loaded Pan Macmillan Onix Source Plugin <---- this one
Whatever is happening there doesn’t seem to take very long on its own, but if it’s generating assets that will be uploaded, that could be adding additional processing time.
I’ll take another look at our code but I’m pretty sure that the formdata log msg is just a stray console.log that shouldn’t be there.
One thing to double check - how does the Gatsby/Netlify cache compare for changes? I assume if data within a component changed (even if that data doesn’t render differently) that would still cause the file to be flagged for upload? Or are you comparing the generated static HTML? I’m wondering if something changed in my stay at the same time as I pushed out this change which would trigger all pages to be deployed.
I’ve just set up another site to test Gatsby cloud but this is also taking a long time to deploy. Are you able to switch off post processing on this site as well please? API ID is e10dff00-a0b3-4679-a080-c666d3cc8d2b
Oh wow I didn’t realize this was a catch all solution now. In the past I remember disabling the forms and still having preprocessing slow the builds down. I guess there’s been some changes since then? Regardless, thanks for pointing that out. I’m sure a lot of people would benefit.
That is a question I haven’t encountered before! Can you share a bit more about why this would be beneficial for you, and what you are hoping to accomplish? This will help me get some actionable info for you.
Additionally, please provide the Netlify site name as well as a link to the form that you are having trouble with.
Our build time has doubled from 10 min to 20 min due to form detection on new pages. We want our decreased build time back! This is for: project-jam-dev
