Gitlab LFS hanging builds

qoouep · February 12, 2020, 8:43am

Hi,

I’m trying to push a change to my site, but it’s hanging on the build. I added lfs hosted on Gitlab in this change, so I suspect that the issue is related to that.

My builds stop at:
12:14:37 AM: Preparing Git Reference merge-requests/88/head

Is there an issue with using Gitlab LFS to host files? I don’t need any of the Netlify LFS features, and I need these files across 3 Netlify sites, so I’d like to stick with Gitlab LFS.

Is it possible to use Gitlab LFS with Netlify sites?

Thanks,

Chris

luke · February 13, 2020, 2:43am

Hi, @qoouep, and thank you for joining our Netlify community site.

If you are using Git LFS (but not Netlify Large Media) an environment variable can be used to the our build system to do a git lfs clone for the repo.

That environment variable is GIT_LFS_ENABLED and setting that to true .

This can be set at “Site Name” > Settings > Build & deploy > Environment in our web app or in the netlify.toml configuration file for the site.

Would you please confirm that environment variable is used or test it if it is not? If it is set and there are still issues, please reply anytime. We’ll be happy to troubleshoot further.

qoouep · February 13, 2020, 6:27am

I was able to get it to work! I have a bit of feedback though.

It’s pretty bad UX for builds to hang instead of surfacing errors. Are there error messages that get swallowed here? Could the site use better messaging here?
Initially I thought it wasn’t working. I was putting my entire assets folder in lfs, and my favicons weren’t loading. It turns out that for some reason specifically the .ico files were not getting pulled in, and were loading as pointer data when I went to /favicon.ico. I was able to fix this for myself by excluding the .ico files from LFS, but this is kind of strange to me. Why was this happening? Will it miss other files?

luke · February 17, 2020, 9:47am

Hi, @qoouep, I’d like to research this further before filing a new issue (as I think it may match an existing issue). Would you please send us a link to the deploy with the error above (and you can just send the deploy id from the end of the URL if you prefer)?

This will tell me if there is a swallowed error we could surface.

Regarding question two about the .ico file, this isn’t one of the file type which is clone in binary format. The defaults as listed here:

github.com

netlify/build-image/blob/xenial/buildbot-git-config#L2


      
          [lfs]
          	fetchinclude = *.jpg,*.png,*.jpeg,*.svg,*.gif,*.pdf,*.mp4,*.bmp,*.webp
          [filter "lfs"]
          	clean = git-lfs clean -- %f
          	smudge = git-lfs smudge -- %f
          	required = true

To get the git lfs clone to include that file extention, an additional environment variable is required, that being GIT_LFS_SMUDGE_FILTER. If you want to include .ico files in addition to the current defaults, this should be set like so (in the Netlify UI only as it won’t work in the repo itself):

GIT_LFS_SMUDGE_FILTER="*.jpg,*.png,*.jpeg,*.svg,*.gif,*.pdf,*.mp4,*.bmp,*.ico"

We do have an open issue to update our documentation to better cover third-party Git LFS configurations like these.

Again, a link to the deploy will help confirm the next steps for question one above.

qoouep · February 18, 2020, 11:14am

Here are a couple:
5e43acc92b1f280008df501a
5e43bef53879f200084b5411

Is there a reason why the files can’t be detected automatically with git lfs ls-files or something?

luke · February 19, 2020, 10:28am

Hi, @qoouep, I did confirm there is no error message being suppressed. I filed a feature request here to not filter the Git LFS tracked files when GIT_LFS_SMUDGE=1 is used:

github.com/netlify/build-image

Don't set a GIT_LFS_FETCH_INCLUDE and always clone the actual content for all Git LFS tracked files

opened 10:12AM - 19 Feb 20 UTC

overlordofmu

Currently the build image only clones the actual file contents for Git LFS track…ed files for file extensions listed here: https://github.com/netlify/build-image/blob/xenial/buildbot-git-config#L2 Quoting: ```text fetchinclude = *.jpg,*.png,*.jpeg,*.svg,*.gif,*.pdf,*.mp4,*.bmp ``` This behavior can be overridden by setting `GIT_LFS_FETCH_INCLUDE ` to a more inclusive list in the build environment with that environment variable. I wonder if we might be better off never filtering and always cloning the any Git LFS files regardless of filetype.

If there are other questions about this, please let us know.