6:11:00 PM: System Error

10:33:03 PM: Skipping functions preparation step: no functions directory set
10:33:03 PM: Caching artifacts
10:33:03 PM: Started saving node modules
10:33:03 PM: Finished saving node modules
10:33:03 PM: Started saving pip cache
10:33:03 PM: Finished saving pip cache
10:33:03 PM: Started saving emacs cask dependencies
10:33:03 PM: Finished saving emacs cask dependencies
10:33:03 PM: Started saving maven dependencies
10:33:03 PM: Finished saving maven dependencies
10:33:03 PM: Started saving boot dependencies
10:33:03 PM: Finished saving boot dependencies
10:33:03 PM: Started saving go dependencies
10:33:03 PM: Finished saving go dependencies
10:33:03 PM: Build script success
10:33:03 PM: Starting to deploy site from 'public'
10:33:04 PM: Creating deploy tree asynchronously
10:33:05 PM: Creating deploy upload records
10:33:11 PM: 6083 new files to upload
10:33:11 PM: 0 new functions to upload
10:34:47 PM: System Error

Incredible support, incredible service :face_with_raised_eyebrow:

hey @pleer-video, are you still seeing this error? I am trying to figure out whether this is a temporary issue or not. If it is still happening, i will figure out what we need to do to get it fixed.

hey @pleer-video - just checking in. If you are still seeing the same problem, please let us know, otherwise we’ll assume the issue resolved itself and we’ll close this thread :slight_smile:

6:00:35 AM: Skipping functions preparation step: no functions directory set
6:00:35 AM: Caching artifacts
6:00:35 AM: Started saving node modules
6:00:35 AM: Finished saving node modules
6:00:35 AM: Started saving pip cache
6:00:35 AM: Finished saving pip cache
6:00:35 AM: Started saving emacs cask dependencies
6:00:35 AM: Finished saving emacs cask dependencies
6:00:35 AM: Started saving maven dependencies
6:00:35 AM: Finished saving maven dependencies
6:00:35 AM: Started saving boot dependencies
6:00:35 AM: Finished saving boot dependencies
6:00:35 AM: Started saving go dependencies
6:00:36 AM: Finished saving go dependencies
6:00:40 AM: Build script success
6:00:40 AM: Starting to deploy site from 'public'
6:00:43 AM: Creating deploy tree asynchronously
6:00:45 AM: Creating deploy upload records
6:01:30 AM: 48257 new files to upload
6:01:30 AM: 0 new functions to upload
6:18:46 AM: Build exited unexpectedly

well, that’s a bummer - we were feeling hopeful we had addressed the issue. we’ll take another look at this next week with the team - it seems possible that this error only occurs when a large amount of files are being uploaded.

Thanks for the answer. Is there a limit on the number of files?

to my knowledge there is no explicit limit, but there very may well be practical limits.

We have been working on this part of the build process and also the logging for a little while now (you can see how the logging error changed from “System Error” to “build exited unexpectedly”.

There is another person here in the forums who also is trying to upload the same high amount of files (in the tens of thousands) at the same time:

(ignore the first half of the thread until we changed his build timeout)

and my best guess is that something with these large volume of files is causing problems, which is why I need to go figure out who to talk to about it (and then find time to troubleshoot).

One question I do have: what kind of deployment workflow are you using? are you just pushing to the repo and then deploying via CI?

I wonder if it would work better if you used the CLI :thinking: it is worth a try if you have the inclination to experiment a bit.

I know this is super frustrating but I promise that we are actively looking at it, it is just proving more complicated to debug than initially expected, and before we can roll out a fix for these unique cases, we need to test and make sure it is safe to use at scale for 800000+ people worldwide…

7:57:26 PM: Skipping functions preparation step: no functions directory set
7:57:26 PM: Caching artifacts
7:57:26 PM: Started saving node modules
7:57:26 PM: Finished saving node modules
7:57:26 PM: Started saving pip cache
7:57:26 PM: Finished saving pip cache
7:57:26 PM: Started saving emacs cask dependencies
7:57:26 PM: Finished saving emacs cask dependencies
7:57:26 PM: Started saving maven dependencies
7:57:26 PM: Finished saving maven dependencies
7:57:26 PM: Started saving boot dependencies
7:57:26 PM: Finished saving boot dependencies
7:57:26 PM: Started saving go dependencies
7:57:26 PM: Finished saving go dependencies
7:57:30 PM: Build script success
7:57:30 PM: Starting to deploy site from 'public'
7:57:33 PM: Creating deploy tree asynchronously
7:57:35 PM: Creating deploy upload records
7:58:31 PM: 6403 new files to upload
7:58:31 PM: 0 new functions to upload
8:00:06 PM: Build exited unexpectedly

mkdir -p public && node --max-old-space-size=8192 ./download.js && mkdir -p public && node --max-old-space-size=8192 ./structure.js
I am running 2 NodeJS, the first is downloading structured XML from the API. And the second parses this XML into JSON files. Because of this, I have 40,000+ JSON files.

5:53:58 PM: Skipping functions preparation step: no functions directory set
5:53:58 PM: Caching artifacts
5:53:58 PM: Started saving node modules
5:53:58 PM: Finished saving node modules
5:53:58 PM: Started saving pip cache
5:53:58 PM: Finished saving pip cache
5:53:58 PM: Started saving emacs cask dependencies
5:53:58 PM: Finished saving emacs cask dependencies
5:53:58 PM: Started saving maven dependencies
5:53:58 PM: Finished saving maven dependencies
5:53:58 PM: Started saving boot dependencies
5:53:58 PM: Finished saving boot dependencies
5:53:58 PM: Started saving go dependencies
5:53:58 PM: Finished saving go dependencies
5:54:02 PM: Build script success
5:54:02 PM: Starting to deploy site from 'public'
5:54:06 PM: Creating deploy tree asynchronously
5:54:08 PM: Creating deploy upload records
5:55:16 PM: 1814 new files to upload
5:55:16 PM: 0 new functions to upload
5:58:07 PM: Build exited unexpectedly

Howdy, and sorry for the long-standing deploy troubles you’re seeing! I spent awhile looking into this via our internal logs, and the reason for that error is still not obvious to me; as you can see from the logs, the build completes succesfully and the problem is in uploading.

So, we have scheduled time with our build system experts tomorrow to get their advice as to how we can get your builds to function reliably.

Thank you for your message.
I’ll attach another log, as you can see, the number of files does not matter, the error is also displayed with a small number.

7:08:26 PM: Skipping functions preparation step: no functions directory set
7:08:26 PM: Caching artifacts
7:08:26 PM: Started saving node modules
7:08:26 PM: Finished saving node modules
7:08:26 PM: Started saving pip cache
7:08:26 PM: Finished saving pip cache
7:08:26 PM: Started saving emacs cask dependencies
7:08:26 PM: Finished saving emacs cask dependencies
7:08:26 PM: Started saving maven dependencies
7:08:26 PM: Finished saving maven dependencies
7:08:26 PM: Started saving boot dependencies
7:08:26 PM: Finished saving boot dependencies
7:08:26 PM: Started saving go dependencies
7:08:26 PM: Finished saving go dependencies
7:08:30 PM: Build script success
7:08:30 PM: Starting to deploy site from 'public'
7:08:34 PM: Creating deploy tree asynchronously
7:08:36 PM: Creating deploy upload records
7:08:47 PM: 65 new files to upload
7:08:47 PM: 0 new functions to upload
7:22:34 PM: Build exited unexpectedly

Hi there and thanks for your patience while we worked with our build team to understand the failure. Unfortunately all we found is that the deploy fails due to running out of memory, which is not expected/usual (I’ve never seen it before, and I’ve debugged well north of 20,000 builds in the past 4 years).

It’s interesting because your site doesn’t seem much bigger than tons of sites that work well:

6:39:18 PM: 6078 new files to upload

is well within the range of site sites we intend to support. Can you confirm that’s as many files as your build generates rather than a very small (<10%) subset? If it’s a huge deploy I could try to talk you through sharding your site or uploads to try to get a successful deploy, but if it isn’t, we may not be able to create a fix in the near term - in which case we’d refund you as much of any past payments you’ve made as we can. I don’t see any charges in your account at present but if you’ve had some I would love to hear about it in that case.

But, I’d love to first hear how many files your build is, and how much space they take up on disk when you build locally, just to get that sense!

Thanks in advance for your help in troubleshooting.

Thank you for trying to understand the problem.

I have about 40,000 json files (~280 MB) + 5,000 html files (~5 MB) = ~300 MB

Is 50,000 static files a lot? What is the limit on the number of files?

There us no specific limit on number of files, but higher numbers use more memory during our upload process. We have many sites with that many (~45k) files, but 500k would be more than I’d expect to “just work”. Thanks for those stats, they’ll inform that bug report.

As far as things you could try to unblock yourself today, since we won’t be able to investigate more deeply in the immediate future: you might consider trying to deploy subsets on separate sites and stitch them together as described here (you’ll use the same repo, but you can proxy to the second site using the pattern described there: [Support Guide] Can I deploy multiple repositories in a single site?). Suggestion: before trying to set up proxying, try simple deploys of half of your content and see if the deploys complete, since that’s a prerequisite for it being worth your time to try that sharding.

Hello. I think I’m going beyond a certain limit. Could you say what are the service limits?

5:46:45 PM: Skipping functions preparation step: no functions directory set
5:46:45 PM: Caching artifacts
5:46:45 PM: Started saving node modules
5:46:45 PM: Finished saving node modules
5:46:45 PM: Started saving build plugins
5:46:45 PM: Finished saving build plugins
5:46:45 PM: Started saving pip cache
5:46:52 PM: Finished saving pip cache
5:46:52 PM: Started saving emacs cask dependencies
5:46:52 PM: Finished saving emacs cask dependencies
5:46:52 PM: Started saving maven dependencies
5:46:52 PM: Finished saving maven dependencies
5:46:52 PM: Started saving boot dependencies
5:46:52 PM: Finished saving boot dependencies
5:46:52 PM: Started saving go dependencies
5:46:52 PM: Finished saving go dependencies
5:46:52 PM: Build script success
5:46:52 PM: Starting to deploy site from 'public'
5:47:57 PM: Creating deploy tree asynchronously
5:48:21 PM: Creating deploy upload records
5:48:28 PM: Failing build: Failed to deploy site
5:48:28 PM: Failed during stage 'deploying site': Failed to execute deploy: Put https://api.netlify.com/api/v1/sites/9f9381ec-9aa8-41cc-ae47-c0fc3f34aa4d/deploys/5ec692c5e3af1b017423b796: context deadline exceeded
5:48:28 PM: Finished processing build request in 8m20.956564722s

Hi, @pleer-video, there is a fifteen minute limit on builds and an additional fifteen minutes permitted for any post processing needed (if any).

Would you please send us a link to this deploy? (Or send us the “deploy id” is the last path segment of the URL when viewing the deploy detail/logs in our web app?)

Also, do you know approximately how many files there were in that site build?

Additional questions are always welcome and we’ll be happy to research the deploy once we know which one it is. :+1:

Thanks for your reply.

5:48:28 PM: Failed during stage ‘deploying site’: Failed to execute deploy: Put https://api.netlify.com/api/v1/sites/9f9381ec-9aa8-41cc-ae47-c0fc3f34aa4d/deploys/5ec692c5e3af1b017423b796: context deadline exceeded

Is this not a link?

Thanks, that was what we were looking for around links. After discussion with our backend team, we probably won’t be able to support deploying that site on our system at present. Sorry I don’t have better news for you today! Should we get the bug we have opened on the problem fixed, we will absolutely notify you so you can confirm our fix and use our platform.