Branch cache fallback behaviour

On the first push of a PR, Netlify takes the cache from the master branch and builds from that. However, after the first push of a PR, all subsequent pushes to that PR will use the branch specific cache. The problem arises when that second push happens quite quickly after the first push. There is a period of time after the first PR build where Netlify has to create the new branch cache and if the second push happens before this is created then it thinks there is a cache miss and then starts the entire build from scratch, fresh clone, install gems, node modules, etc.

I’d like to see the behaviour change to keeping the first lookup for the branch cache but if that misses then first revert to looking for the master cache before finally falling back to building from scratch. Doing that seems like a reasonable solution and even if your first PR added a node module or 2 for example then you are just reinstalling those again on a branch cache miss rather than everything.

We currently experience a number of cache misses every day on our builds and this would pretty much eliminate the problem. I’m sure a lot of others would benefit from this as well.

Thanks

Thanks for that feedback, Jeffrey!

I’ll get our existing feature request updated around that misbehavior; hadn’t anticipated someone managing to trigger two builds before we instantiated the cache so hadn’t anticipated that problem.

How are you managing to do that, though - what’s your flow for triggering a build on that PR? It’s our intention that we wouldn’t start that second PR build until the first was finished, which would obviate that need/situation/

Hi @fool, I think this is occurring in our case because of the size of our site and the time it takes to update the cache following a build. When one build finishes, we get a notification in the logs saying “Site is live” and at that point the second build will start. However, on the first build there is roughly another 2 minutes or so before the final notification gets logged saying “Finished processing build request in …” which I believe is when the branch cache has been updated. Therefore it’s in this window where missed cache occurs - the second build has started but the first build hasn’t updated the cache yet.

We have content editors pushing content from Netlify CMS very regularly and some are pretty Save happy :smiley: which means this situation is happening quite a lot.

Ah, wow, your cache must be huge if it takes 2 minutes to handle! Any chance you could reduce the cache (e.g. prune old commits from your existing repo into a new repo with less history to transfer around?) That would certainly help avoid this…

However, I’ll get a bug filed on this for our build team to consider. I’ll post back in this thread if things change but no promises, since I won’t be doing the debugging or writing the code (or deciding the team should look into this) - I ask on our behalf but there are a lot of things already planned and bugs that affect more customers that are likely to be prioritized before it.

Ha, yeah the cache was 2.4GB! We just did some work this week to reduce the repo size by using git-filter-repo to remove old history from the pack files that shouldn’t have been there. Quite a task doing that without errors but it took the repo size down from around 1.7GB to roughly 700MB and the Netlify cache is now 1.3GB. We are also moving over 3000 images out of the repo next week and onto a cloud service so that will take the size down further.

However, there will still be a gap between the logs saying “Site is live” and “Finished processing build request in …” so if another build is ready to begin straight away then the issue will still happen.

Thanks for filing the bug and I understand about the planning. I’ll keep an eye on the thread :+1: