[Support Guide] How can I optimize my Netlify build time?

Is it possible to view this data in a similar way to how we view Netlify Analytics. If we could see how much minutes are being used per day in a 1 month period it would be really helpful.

Yesterday, I made some changes to my project decreasing build times from 10 minutes down to 4 minutes. I also modified some settings on netlify like disabling pull request deploy previews and such. Nonetheless, it will be really hard to understand how all these changes impact my build times if all I can see is a number for the entire month.

The data I was referring to:

If we had a chart i could tell after a week hey my total build times have decreased a lot or I still need to make more changes. I could tell if my build times decreased enough to re-enable a feature like deploy previews. One number for the total of a month isn’t enough in my opinion.

One of the best thing has been how this encourages us to create better projects by improving our build times. However, it’s really hard to determine how effective that change is over a period of time.

2 Likes

We’re working on it :wink: Something more granular should be coming soon and we’ll announce it in this thread.

2 Likes

Is it possible to enhance this a little bit please? This is a really good feature but I wish we could have a setting in the Dashboard where we could add our own keywords. In my opinion, this is going to be a massive impact on build times for projects like mine.

1. Dependabot
The bot makes quite a few PR into a project. If you have deploy previews enabled, your build times are going to be getting destroyed. Even when you have it disabled, merging these PRs can cause about 2-4 builds to be run. This certainly needs to be enhanced.

Today I had about 12 PR from the bot. I merged as many as i could as fast as I could until some of them had conflicts preventing merging. While waiting to have the bot resolve these conflicts netlify builds completed 1 build and skipped a few and began building the latest one. Once dependabot fixed the conflicts I merged them as well and once again netlify created a third build.

All this happened while I was rushing to merge them. Normally, I would just come back later and merge any remaining PRs. This is going to be causing anywhere between 2-4 builds for normal users. Something else that I have not yet considered is what happens when you have more than 1 build capacity. If i were to have 3 build capacity this might become a nightmare haha.


2. Netlify CMS
My project also incorporates netlify CMS and maintainers can write guides on the project. This includes uploading images and creating guides. The issue is that uploading a single image can trigger an entire build as well. This becomes really problematic even if you have not merged that PR because netlify CMS edits the repo directly with uploaded images.


If it was possible to somehow control what keywords to look for in the settings we could limit a lot of these builds from being triggered.

Great suggestion, but I don’t think we have immediate plans to implement it in the near term for the general use case of commits where you specify the commit message, which I understand is NOT your use case. I wonder if @erquhart could speak to the ability to customize the CMS commit message that the CMS uses?

I have nonetheless filed a feature request for it, and linked this thread to it so we can respond here in case we do.

One of our maintainers just opened an issue for this on the Netlify CMS repo, very doable.

1 Like

Can you check if the ignore setting in netlify.toml might work for you?

You could try something like this: git log -1 --pretty=%B | grep dependabot.
Git log has all kinds of info about a commit, so you could also check who committed.

4 Likes

Turns out custom commit messages are already available as a beta feature:

backend:
  commit_messages:
    create: 'Create {{collection}} “{{slug}}”'
    update: 'Update {{collection}} “{{slug}}”'
    delete: 'Delete {{collection}} “{{slug}}”'
    uploadMedia: '[skip ci] Upload “{{path}}”'
    deleteMedia: '[skip ci] Delete “{{path}}”'
3 Likes

I had started a separate thread for this issue, but I guess this would be a more appropriate place for it.

If I’m not getting something wrong, netlify is making many unnecessary builds, at least for my project. Please take a look a this screenshot of my builds dashboard:

There are three different builds for the same commit (note the highlighted commit messages).

The first one refers to the actual commit, but the two following ones correspond to changes in the gitlab MR (such as changing the MR’s description or removing its WIP status). How can I avoid this?

@LGenzelis, I followed up in the there topic with an answer here.

To summarize, the webhook for MRs will be “triggered when a merge request is created/updated/merged”. There isn’t an option for just “merged” and this setting is at GitLab, not Netlify, so we cannot change this behavior. It might be helpful to file a feature request at GitLab asking for more fine grained control of the webhooks there.

I would love an option to opt-in to branch builds instead of opting out. (IE a [build] in a commit comment).

We find the branch deploys extremely useful for sharing progress and getting sign off from other teams before code is merged, but many branches there’s no need to deploy, and many commits also don’t need to be built. having a way that we can easily and from the command line let devs deploy to branchname--us.netlify.com would work really well to limit unnecessary use of build minutes.

Is anything like that possible out of the box?

Hi, @thedamon. We want to make certain we understand the use cases and requirements for the new feature request before we file it.

When you say “from the command line” is this a reference to Netlify’s CLI tool or a Git commit/push?

Regarding only building specific branches, that can be configured already on a per-site basis. The options are:

  • build only the production branch
  • build all branches
  • build production and selected branches only

Do those options meet the “many branches there’s no need to deploy” requirement? If not, what behavior would better meet the requirement?

Regarding the opt-in (as opposed to the current opt-out), how would you see that working? Would there be an “opt-in only” option/checkbox in the site settings which prevents builds unless “[build]” (or some other string) is included in the commit message? Should the opt-in string be user configurable?

Please feel free to include any additional details which help to clarify the request and we look forward to your reply.

hey @luke thanks so much for being in touch.

“build production and selected branches only” is close, but selecting which branches becomes the pain point. In order to conserve minutes we want to avoid as many deploys as we can… but we wouldn’t want to need to go into the netlify admin to set it up.

Generally, the perfect flow would be to have a command line option to choose to also trigger a netlify branch deploy (to branchname--me.netlify.com whenever pushing new commits (mainly to reduce the overhead in using this feature).

Under the covers this could be something like using an API to turn on branch deploys for a branch, push, and then turn off branch deploys for the branch immediately after the push. It’s possible that something like that already exists in the CLI tool… I haven’t checked that out yet!

[build ci] (just to follow the skip-ci pattern) as a potential flag in a commit message would be awesome to be able to use ( maybe not necessary given the stuff above )… I’m always happy to be able to configure stuff, but I don’t think it’s a big deal here.

Alternatively, you could set up a build hook for a specific branch and then trigger that hook whenever you need to. You could probably automate when to do so using GitHub Actions or something similar.

Would that work for you?

Trying to skip all dependabot PRs I user @marcus’ suggestion

[build]
  # ignore all PRs opened by `dependabot` (with last commit made by `dependabot`)
  ignore = 'git log -1 --pretty=%B | grep dependabot'

Netlify is triggering the ignore flag correctly, on a commit hash which should result in a match (and a 0 return code), however it still runs the build. Is there something I’m missing?

12:33:25 AM: Found Netlify configuration file. Overriding site configuration
12:33:25 AM: Detected ignore command in Netlify configuration file. Proceeding with the specified command: 'git log -1 --pretty=%B | grep dependabot'
12:33:25 AM: Starting build script

EDIT: I relinked the Github repository as specified in netlify.toml documentation, and now the ignore task seems to run. However I get an error fatal: ambiguous argument '|': unknown revision or path not in the working tree.

11:13:37 AM: Detected ignore command in Netlify configuration file. Proceeding with the specified command: 'git log -1 --pretty=%B | grep dependabot'
11:13:37 AM: fatal: ambiguous argument '|': unknown revision or path not in the working tree.
11:13:37 AM: Use '--' to separate paths from revisions, like this:
11:13:37 AM: 'git <command> [<revision>...] -- [<file>...]'
11:13:37 AM: Starting build script
1 Like

Hmm, that’s quite odd - I just pasted that into my shell and it worked well.

Just for grins, could you try making a shell script called “ignore.sh” in the root of your base directory (if any), and using as the command sh ./ignore.sh ? It should look like this:

#!/bin/sh

git log -1 --pretty=%B | grep dependabot

Let me know how it goes!

@laura awesome, thanks!

1 Like

Thanks!

Hmm, we started getting intermittent Bad Gateway 502 errors on the same date that you made the change. It happens during the CDN diffing step:

  • CDN diffing files…
    › Warning: TextHTTPError: 502
    › Warning:
    › TextHTTPError: Bad Gateway

    TextHTTPError: Bad Gateway
    at NetlifyAPI.createSiteDeploy (/app/landing/node_modules/netlify/src/open-api/index.js:173:13)
    at async module.exports (/app/landing/node_modules/netlify/src/deploy/index.js:77:16)
    at async NetlifyAPI.deploy (/app/landing/node_modules/netlify/src/index.js:108:12)
    at async DeployCommand.run (/app/landing/node_modules/netlify-cli/src/commands/deploy.js:158:17)
    at async DeployCommand._run (/app/landing/node_modules/@oclif/command/lib/command.js:44:20)
    at async Config.runCommand (/app/landing/node_modules/@oclif/config/lib/config.js:151:9)
    at async Main.run (/app/landing/node_modules/@oclif/command/lib/main.js:21:9)
    at async Main._run (/app/landing/node_modules/@oclif/command/lib/command.js:44:20)

Could it be related?

It looks like our team is bumping up against the threshold allotted for Open Source projects. We build and run e2e tests using CircleCI before accepting a pull request; is it possible, using the Netlify API, to publish a “Deploy Preview” from Circle CI?

The hope being that there’s no reason to build our application multiple times on different platforms and that this could save us some build minutes.

If not, CircleCI has a “Manual Approval” workflow that we use to gate expensive tasks. Is there a mechanism for kicking off a deploy preview manually from a GitHub status check?

@fool @perry Yikes, this 502 error is happening a lot, I don’t see it in the logs before Feb 10. Is there any reason that they could be connected?

Doesn’t seem like 502’s would be related, no, @salmonax. 502 usually means that you have a very large site that isn’t being synced fast enough. You could try adding a --timeout 900 flag with your CLI usage to see if it helps.