Limit Concurrent Builds

Hi there we are using a PRO PLAN with netlify, we have 3 concurrent builds in our plan, but more can be used by paying extra amount (40$ probably).

We have to build around 250 sites at a time, system at times builds 10+ websites at one time, which sends extra unrequired load to our server.
We do not want concurrent builds to be more than 3.

Is there an option in admin settings where we can restrict the concurrent builds to 3 regardless of how many website build requests do we send?

Hi, @mhaziq. The concurrent builds setting doesn’t change automatically. If you send 250 builds at the same time on a Pro plan team, by default only three will build at a time. The other 247 builds will queue up and build only using three concurrent builds at a time.

You can increase the concurrency above three manually but it won’t change unless you manually do so yourself. However, I see you wrote this:

We have to build around 250 sites at a time, system at times builds 10+ websites at one time, which sends extra unrequired load to our server.

That should not happen if the concurrency is only set to 3. Would you please send me a link to your Pro plan team? (I didn’t find it under the email address you used for this forum.)

You can post that information publicly or you can private message (PM) that to one of our support staff. I’ve confirmed that PMs are enabled for your community login.

Hi, @mhaziq. I did get the private message with the account details and the rest of the support team can see that information now as well.

I’m showing the builds for that team do queue up when there are more than three at once.

For example, I looked the group of builds that triggered at 8:00 pm PST (04:00 UTC) today. All builds are triggered but they don’t build in parallel. In one build I see this:

8:00:41 PM: Build ready to start
8:00:42 PM: build-image version: 53b83b6bede2920f236b25b6f5a95334320dc849
8:00:42 PM: build-image tag: v3.6.0
8:00:42 PM: buildbot version: a706ec7a557bcc28584843816a376a10c08955ca
8:00:42 PM: Building without cache
8:00:42 PM: Starting to prepare the repo for build
8:00:43 PM: No cached dependencies found. Cloning fresh repo

In another I see this:

8:00:41 PM: Waiting for other deploys from your team to complete
8:11:33 PM: Build ready to start
8:11:34 PM: build-image version: 53b83b6bede2920f236b25b6f5a95334320dc849
8:11:34 PM: build-image tag: v3.6.0
8:11:34 PM: buildbot version: a706ec7a557bcc28584843816a376a10c08955ca
8:11:34 PM: Building without cache
8:11:34 PM: Starting to prepare the repo for build
8:11:35 PM: No cached dependencies found. Cloning fresh repo

This second build was paused until other builds had finished. It did queue for 10 minutes and 52 seconds before building.

Currently our system creates a “build object” when the request for a new build is received. If there is not sufficient build capacity for the concurrency setting, builds are queued until other builds have completed. However, the build object is created and any related webhooks for a new build starting are triggered.

Would you please confirm if I understand the problem and solution required?

I think you are asking for this change:

  • Our system should not create the build object at all until capacity is available.

I’m guessing that is because the webhook that triggers when the build object is created triggers activity on your systems and you want the webhook not to trigger until the build object is no longer queued.

Is that an accurate understanding of the issue?

If so, I can enter a feature request for this but I cannot promise if or when that feature request will be available. Even if we enter the feature request we can still look for workaround or other solution in the meantime. I just want to be sure I understand the problem correctly before looking for other solutions.

If a webhook is to blame, the usual recommendation would be to change that to a webhook that only triggers when a deploy is successful, not when it starts. However, I still don’t know where the load is coming from. Is it coming from a webhook?

Also, if I am not understanding the issue correctly, would you please correct me and explain more about what you want changed or what problem is occurring?

If there are other questions for us, please include them at any time and we look forward to your reply.

Hi Luke the latest builds which you have looked in our account were good, this ticket was created probably before that, If you look at the screenshot I sent to you in person, you ll notice that it shows around 8 sited in “Building” status whereas normally all of them should have status “Enqueued: Awaiting Capacity” except the 3 of them(which fall in our capacity).

The issue which I have reported, occurred around 2 week ago, its not happening now, I took a screenshot gladly at that time, which you can see.

Hi, @mhaziq. I cannot tell from the screenshot you sent what date or timezone is being shown. It shows only this for the timestamp:

screenshot

The timestamp is timezone specific and there are 304 pages of deploys to check. In short, I don’t yet have enough information to track down these deploys. What was the date and timezone when that screenshot was made?

Similarly, if you would send a link to one of those deploys (or send the deploy id for the deploy), that will also help me find the deploys in question.