Constant and frustrating build issues with LFS

Hello,

I’ve lost my sanity trying to deal with LFS in Netlify and need help. Frequently encountered problems during builds #8 told me to reach out if I was still having issues. I think I might be in that 1%.

I’m building a Gatsby site from GitLab using their LFS implementation for images and fonts. When developing locally I’m able to work normally. My files are pulled from LFS and the file pointers are replaced with the real files. My local production builds work just fine.

When I started using Netlify to deploy my site this week I noticed that webpack was inlining my fonts (as base64 url blobs) into the site, but when decoded those blobs only contained the content of the LFS file pointers. After a lot of troubleshooting and trial and error I discovered that Netlify handles LFS strangely.

GIT_LFS_ENABLED=true is required to fix build cache errors (which I experienced), but that just uses git lfs clone instead of git clone. git lfs clone is deprecated and LFS is handled internally by git clone.

❯ git lfs clone git@gitlab.com:johnrichter/personal-website test-lfs-clone
WARNING: 'git lfs clone' is deprecated and will not be updated with new flags from 'git clone'

'git clone' has been updated in upstream Git to have comparable speeds to 'git lfs clone'.

My local git versions are

  • git version 2.24.0
  • git-lfs/2.8.0 (GitHub; darwin amd64; go 1.12.7)

I was able to figure out that only the font files (woff2, woff, eot, and ttf) would not clone successfully with LFS.

$ git lfs fetch --all; git lfs checkout; cat /opt/build/repo/.git/lfs/logs/*
fetch: 100 object(s) found, done
fetch: Fetching all references...
batch request: missing protocol: ""
error: failed to fetch some objects from ''
...
Skipped checkout for "src/assets/fonts/Exo_300.eot", content not local. Use fetch to download.
Skipped checkout for "src/assets/fonts/Exo_300.ttf", content not local. Use fetch to download.
Skipped checkout for "src/assets/fonts/Exo_300.woff", content not local. Use fetch to download.
Skipped checkout for "src/assets/fonts/Exo_300.woff2", content not local. Use fetch to download.

The first issue I’ve found is with Git. batch request: missing protocol: "" and error: failed to fetch some objects from '' resolve to issues with git lfs not having SSH support and the dependence that LFS has on the remote being origin. I was able to add origin by running git remote add origin https://gitlab.com/johnrichter/personal-website.git before gatsby build, but that alone didn’t fix issues since it happened after the initial clone. After adding git lfs fetch --all && git lfs checkout I was able to finally able to get LFS to load all files correctly.

My final build command was git remote add origin https://gitlab.com/johnrichter/personal-website.git && git lfs fetch --all && git lfs checkout && gatsby build.

I thought I was home free, but now I’m running into issues with subsequent builds that are saying that remote origin already exists. This does not happen if I manually build without the cache.

Git LFS enabled
Preparing Git Reference merge-requests/15/head
Error adding remote: fatal: remote origin already exists.
Failing build: Failed to prepare repo

This support question hinted at writing a script that adds origin if it doesn’t exist before the build and then removes origin after the build to avoid the remote being configured for the next build from the Netlify cache, but its not working. I’ve tried to run this script directly as the build command and as a yarn preinstall command, but nothing works. Using preinstall did show me that my build script environment needs to be set carefully. When running the script as a normal build script I tried using set -ex bash ..., but that didn’t work. Perhaps I need to do that within the script.

At this point I’m contemplating on doing my builds and deployments within GitLab CI via Netlify CLI because I think it’d be easier. I’d rather use Netlify CD to do this bercause of the ease of automatic deploy previews though!

The repo is here. My Netlify site is jrichter-io.netlify.com. Links to the builds for troubleshooting are in my follow up comment because I’m limited to 6 links for new topics in this forum.

Thanks for reading to the end. I’d really really appreciate help. I’m not sure what to do. Maybe I’ll pull down the build image, read its source, and try to rapidly test solutions locally in Docker.

Links to build logs:

More links to build logs. This 6 link limit is really annoying.

Good news everyone!

I found build-image#175 which described my exact issue with fonts not being loaded from LFS. It suggested using GIT_LFS_FETCH_INCLUDE to override the lfs.fetchinclude filter set within the build image.

I had no idea this config existed and I’m going to guess that lfs.fetchinclude is being used for large file protection. I thought this was a Git env variable at first, but after some digging in the git lfs repo it seems that it’s actually a Netlify-specific env variable. Why this isn’t documented somewhere has been beyond incredibly frustrating. I’ve spent over 24 hours and over 200 build minutes trying to figure that all I needed was to override this config setting.

My build command is now simply gatsby build with GIT_LFS_ENABLED=TRUE and GIT_LFS_FETCH_INCLUDE=* in the UI.

18


I’m still having issues with the cache though. All builds using the cache fail with Error adding remote: fatal: remote origin already exists.. I’m not doing any git repo manipulation anymore and I haven’t figured out how or found the source code for the build system’s repo restoration from cache. It is clearly trying to update the remote.

An example build

By the way, these “new user limits” are incredibly frustrating. I just want to give as much detail to the problem as possible to make troubleshooting easier for y’all and I’m forced to do that with many comments on my own thread.

21

Oh, the irony.

@johnrichter, thank you for the detailed information and welcome to our Netlify community site. Sharing all of those discoveries here will (hopefully) be quite beneficial to other people searching community for similar issues.

Back to the issue you are asking about! The build linked to most recently failed because an attempt to update the repo was happening using a cached repo clone which was now out of date.

I retried that same deploy preview with “Retry deploy” button on the build page itself and used the option “Clear cache and deploy site” when confirming. This new deploy was successful which can be checked here:

https://app.netlify.com/sites/jrichter-io/deploys/5dd779143c9166a62089fa31

Would you please confirm that that version of the deployed site has all required assets (like the embedded fonts) being served correctly now? It looks correct to me but I could be missing something. I can confirm the font warnings in the javascript console do not occur on the deploy above (but did see in the non-working deploys).

Unfortunately, I do believe this workaround of clearing the build cache will be needed for any deploys with a cached version of the repo (meaning any deploys for a branch which has been deployed at Netlify before the repo origin change was made). However, this should only be required once per branch and they should continue to work correctly after this.

Would you please try clearing the cache and redeploying for any other branches still getting the deploy error about the origin? If this doesn’t work and/or if there are other questions, please let us know with follow-up post here.

Finally, I also agree the[new user limits on links in posts (which are a build in part of the software our community site uses) are often frustrating. They are unfortunate requirement because of the spam/bot traffic which target sites with a comments/posts feature. (In other words, spammers are why we can’t have nice things. :imp: )

Again, though, we do appreciate you sharing all of this hard work and we’re happy to troubleshoot further if issues persist.

1 Like

@johnrichter, I also skipped you ahead to “trust level 3” since I’m sure your not a spammer. :slight_smile:

1 Like

You’re welcome and thanks for the bump to TL3! I’ve heard of Discourse, but never actually used it before so I appreciate the link to their docs. Its a shame that some people have to ruin it for the rest of us :frowning:


Your deploy looks good :+1: All the fonts are loading correctly and the rest of the site is good. What’s interesting is that I’ve been running the “Clear cache and deploy site” option in the UI to retry builds and achieve successful deployments while this has been happening.

I just ran some tests with a new branch and merged that branch into one with an existing one (develop/v2) just to check all cases since I did the cache clearing yesterday evening. The new branch added new font files in LFS and uses those new fonts throughout the site. Everything built correctly and the deploy previews look good!

All with no git issues. As a bonus, now that the Netlify cache is working correctly with my repo, my Gatsby caching plugin can now cache optimized images and assets between builds. They went from 5-7m down to 1-3m! I’ll have to keep an eye on my cache size over time and reset it if it gets too large, but it’s a huge win in my book. When I first started this on early this week my builds were hitting the 15m timeout… Pre-optimizing all of my images was also a big win.

Thanks for the help @luke!

If you do notice issues with the cache growing over time, please feel free to follow-up here (or in a new topic if you prefer). Also, thank you for follow-ing up to let us know you got this working and what your results are so far.

1 Like