Netlify API file uploads: undocumented concurrency limits

styxlab · October 13, 2020, 8:10am

I am using the Netlify API endpoint PUT /deploys/${deployId}/files/${path} for uploading files with a concurrency of 15 as suggested by the API js-client example:

github.com

netlify/js-client/blob/9915239d6931c49416feda94dfc013eec7e0ff1a/src/deploy/index.js#L23


      
          module.exports = async (api, siteId, dir, opts) => {
            opts = Object.assign(
              {
                fnDir: null,
                configPath: null,
                draft: false,
                message: undefined, // API calls this the 'title'
                tmpDir: tempy.directory(),
                deployTimeout: 1.2e6, // local deploy timeout: 20 mins
                concurrentHash: 100, // concurrent file hash calls
                concurrentUpload: 15, // Number of concurrent uploads
                filter: defaultFilter,
                syncFileLimit: 100, // number of files
                maxRetry: 5, // number of times to retry an upload
                statusCb: () => {
                  /* default to noop */
                  /* statusObj: {
                      type: name-of-step
                      msg: msg to print
                      phase: [start, progress, stop],
                      spinner: a spinner from cli-spinners package

However, it seems that my account only allows a concurrency of 5, at least that’s what the header response is reporting:

    'x-concurrent-limit': [ '5' ],
    'x-concurrent-remaining': [ '0' ],
    'x-concurrent-status': [ 'checked' ],

Unfortunately, this limits leads to upload errors with error code 429. I found some documentation about rate limits, but not on concurrency limits. Furthermore, the above mentioned Netlify example code suggest a concurrency limit of 15 instead of 5. Here are my questions:

Can you confirm that a concurrency limit of 5 is a firm limit for file uploads?
Is there a way to increase this limit to e.g. 15?

Thanks a lot for looking into this issue,
– styxlab

jen · October 18, 2020, 12:08am

Hey @styxlab,
You’re absolutely right about this. It is 5, and we implemented that rate limit recently because we were seeing a lot of abusive clients. We should definitely get this updated in the js-client. At this time, there’s no way to increase the limit, but we can post again here if that changes.

jen · October 21, 2020, 4:19pm

Closing the loop here: we’ve updated that concurrency limit
https://github.com/netlify/js-client/blob/master/src/deploy/index.js#L23

And I misspoke when I said there was no way to get more concurrent uploads. The concurrency limit is 10 on paid plans, including the next plan level up from Starter which is Pro ($19/month). Not sure whether 5 more concurrent builds is worth that for you, but wanted to be sure to share the correct info!

Thanks again for flagging the discrepancy so we could get that fixed.