`git push` and `git lfs push origin master` keeps failing

I keep running git push on my GitHub repository with Netlify Large Media enabled, but the command fails with multiple messages saying:

batch response: Expected json type, got: "text/plain; charset=utf-8"

I was able to incrementally push 80% of lfs tracked files up to my large media object while getting this error message. Now it stopped working and every time I run git push I get:

Uploading LFS objects:   0% (0/100), 0 B | 0 B/s, done.
batch response: Expected json type, got: "text/plain; charset=utf-8"
batch response: Expected json type, got: "text/plain; charset=utf-8"
batch response: Expected json type, got: "text/plain; charset=utf-8"
...
batch response: Expected json type, got: "text/plain; charset=utf-8"
batch response: Expected json type, got: "text/plain; charset=utf-8"
batch response: Expected json type, got: "text/plain; charset=utf-8"
error: failed to push some refs to 'git@github.com:<ORG>/<REPO_NAME>.git'

I was wondering if others are also experiencing similar issues.

If it helps, I got this log once when my git push failed:

git-lfs/2.9.2 (GitHub; windows amd64; go 1.12.7; git 0274d856)
git version 2.25.0.windows.1

$ git-lfs pre-push origin git@github.com:[GIT_USER]/[REPO].git
batch response: Expected json type, got: "text/plain; charset=utf-8"
batch response: Expected json type, got: "text/plain; charset=utf-8"
batch response: Expected json type, got: "text/plain; charset=utf-8"
batch response: Expected json type, got: "text/plain; charset=utf-8"
batch response: Expected json type, got: "text/plain; charset=utf-8"
batch response: Expected json type, got: "text/plain; charset=utf-8"
batch response: Expected json type, got: "text/plain; charset=utf-8"
batch response: Expected json type, got: "text/plain; charset=utf-8"
batch response: Expected json type, got: "text/plain; charset=utf-8"
batch response: Expected json type, got: "text/plain; charset=utf-8"
batch response: Expected json type, got: "text/plain; charset=utf-8"
batch response: Expected json type, got: "text/plain; charset=utf-8"
batch response: Expected json type, got: "text/plain; charset=utf-8"
batch response: Expected json type, got: "text/plain; charset=utf-8"
batch response: Fatal error: Server error: https://[SITE_ID].netlify.com/.netlify/large-media/objects/batch

Server error: https://[SITE_ID].netlify.com/.netlify/large-media/objects/batch
github.com/git-lfs/git-lfs/errors.Errorf
	d:/a/git-lfs/git-lfs/errors/errors.go:69
github.com/git-lfs/git-lfs/lfshttp.defaultError
	d:/a/git-lfs/git-lfs/lfshttp/errors.go:128
github.com/git-lfs/git-lfs/lfshttp.(*Client).handleResponse
	d:/a/git-lfs/git-lfs/lfshttp/errors.go:51
github.com/git-lfs/git-lfs/lfshttp.(*Client).DoWithRedirect
	d:/a/git-lfs/git-lfs/lfshttp/client.go:311
github.com/git-lfs/git-lfs/lfsapi.(*Client).doWithCreds
	d:/a/git-lfs/git-lfs/lfsapi/auth.go:101
github.com/git-lfs/git-lfs/lfsapi.(*Client).doWithAuth
	d:/a/git-lfs/git-lfs/lfsapi/auth.go:67
github.com/git-lfs/git-lfs/lfsapi.(*Client).DoWithAuth
	d:/a/git-lfs/git-lfs/lfsapi/auth.go:25
github.com/git-lfs/git-lfs/lfsapi.(*Client).DoAPIRequestWithAuth
	d:/a/git-lfs/git-lfs/lfsapi/auth.go:56
github.com/git-lfs/git-lfs/tq.(*tqClient).Batch
	d:/a/git-lfs/git-lfs/tq/api.go:74
github.com/git-lfs/git-lfs/tq.Batch
	d:/a/git-lfs/git-lfs/tq/api.go:40
github.com/git-lfs/git-lfs/tq.(*TransferQueue).enqueueAndCollectRetriesFor
	d:/a/git-lfs/git-lfs/tq/transfer_queue.go:520
github.com/git-lfs/git-lfs/tq.(*TransferQueue).collectBatches.func1
	d:/a/git-lfs/git-lfs/tq/transfer_queue.go:432
runtime.goexit
	C:/Go1.12.7/src/runtime/asm_amd64.s:1337
Fatal error
github.com/git-lfs/git-lfs/errors.newWrappedError
	d:/a/git-lfs/git-lfs/errors/types.go:198
github.com/git-lfs/git-lfs/errors.NewFatalError
	d:/a/git-lfs/git-lfs/errors/types.go:242
github.com/git-lfs/git-lfs/lfshttp.(*Client).handleResponse
	d:/a/git-lfs/git-lfs/lfshttp/errors.go:75
github.com/git-lfs/git-lfs/lfshttp.(*Client).DoWithRedirect
	d:/a/git-lfs/git-lfs/lfshttp/client.go:311
github.com/git-lfs/git-lfs/lfsapi.(*Client).doWithCreds
	d:/a/git-lfs/git-lfs/lfsapi/auth.go:101
github.com/git-lfs/git-lfs/lfsapi.(*Client).doWithAuth
	d:/a/git-lfs/git-lfs/lfsapi/auth.go:67
github.com/git-lfs/git-lfs/lfsapi.(*Client).DoWithAuth
	d:/a/git-lfs/git-lfs/lfsapi/auth.go:25
github.com/git-lfs/git-lfs/lfsapi.(*Client).DoAPIRequestWithAuth
	d:/a/git-lfs/git-lfs/lfsapi/auth.go:56
github.com/git-lfs/git-lfs/tq.(*tqClient).Batch
	d:/a/git-lfs/git-lfs/tq/api.go:74
github.com/git-lfs/git-lfs/tq.Batch
	d:/a/git-lfs/git-lfs/tq/api.go:40
github.com/git-lfs/git-lfs/tq.(*TransferQueue).enqueueAndCollectRetriesFor
	d:/a/git-lfs/git-lfs/tq/transfer_queue.go:520
github.com/git-lfs/git-lfs/tq.(*TransferQueue).collectBatches.func1
	d:/a/git-lfs/git-lfs/tq/transfer_queue.go:432
runtime.goexit
	C:/Go1.12.7/src/runtime/asm_amd64.s:1337
batch response
github.com/git-lfs/git-lfs/errors.newWrappedError
	d:/a/git-lfs/git-lfs/errors/types.go:198
github.com/git-lfs/git-lfs/errors.Wrap
	d:/a/git-lfs/git-lfs/errors/errors.go:74
github.com/git-lfs/git-lfs/tq.(*tqClient).Batch
	d:/a/git-lfs/git-lfs/tq/api.go:77
github.com/git-lfs/git-lfs/tq.Batch
	d:/a/git-lfs/git-lfs/tq/api.go:40
github.com/git-lfs/git-lfs/tq.(*TransferQueue).enqueueAndCollectRetriesFor
	d:/a/git-lfs/git-lfs/tq/transfer_queue.go:520
github.com/git-lfs/git-lfs/tq.(*TransferQueue).collectBatches.func1
	d:/a/git-lfs/git-lfs/tq/transfer_queue.go:432
runtime.goexit
	C:/Go1.12.7/src/runtime/asm_amd64.s:1337

Current time in UTC: 
2020-03-03 06:41:23

ENV:
LocalWorkingDir=path/to/working-dir
LocalGitDir=path/to/git-dir
LocalGitStorageDir=path/to/git-storage-dir
LocalMediaDir=path/to/media-dir
LocalReferenceDirs=
TempDir=path/to/temp-dir
ConcurrentTransfers=3
TusTransfers=false
BasicTransfersOnly=false
SkipDownloadErrors=false
FetchRecentAlways=false
FetchRecentRefsDays=7
FetchRecentCommitsDays=0
FetchRecentRefsIncludeRemotes=true
PruneOffsetDays=3
PruneVerifyRemoteAlways=false
PruneRemoteName=origin
LfsStorageDir=path/to/lfs-storage-dir
AccessDownload=basic
AccessUpload=basic
DownloadTransfers=basic,lfs-standalone-file
UploadTransfers=basic,lfs-standalone-file
GIT_EXEC_PATH=path/to/git-exec
GIT_PREFIX=

I’d be happy to share the SITE_ID on DM with the staff.

This is a sample of the type of responses I get from Netlify Large Media servers when I run GIT_CURL_VERBOSE=1 git push:

12:02:05.557263 trace git-lfs: tq: sending batch of size 100
12:02:05.558263 trace git-lfs: api: batch 100 files
12:02:05.558263 trace git-lfs: creds: git credential cache ("https", "[SITE_ID].netlify.com", "")
12:02:05.559263 trace git-lfs: Filled credentials for https://[SITE_ID].netlify.com/.netlify/large-media
12:02:05.559263 trace git-lfs: HTTP: POST https://[SITE_ID].netlify.com/.netlify/large-media/objects/batch

> POST /.netlify/large-media/objects/batch HTTP/1.1
> Host: [SITE_ID].netlify.com
> Accept: application/vnd.git-lfs+json; charset=utf-8
> Authorization: Basic * * * * *
> Content-Length: 8914
> Content-Type: application/vnd.git-lfs+json; charset=utf-8
> User-Agent: git-lfs/2.9.2 (GitHub; windows amd64; go 1.12.7; git 0274d856)
>
{
   "operation":"upload",
   "objects": [
      {
         "oid": "[HASH_VALUE]",
         "size": [SIZE]
      }, 
      ...
   ],
   "transfers":[ "lfs-standalone-file", "basic" ],
   "ref": { "name": "refs/heads/master" }
}

12:02:05.675270 trace git-lfs: HTTP: 200

< HTTP/2.0 200 OK
< Content-Length: 1346
< Age: 0
< Cache-Control: public, max-age=0, must-revalidate
< Content-Type: text/plain; charset=utf-8
< Date: Tue, 03 Mar 2020 20:02:05 GMT
< Server: Netlify
< X-Bb-Proxy: https://lm.services.netlify.com/lfs/objects/batch
< X-Bb-Proxy-Version: v2
< X-Nf-Request-Id: 02187845-bfb4-4242-ad90-b7c7169d9390-660364
< X-Robots-Tag: noindex
<

I deleted my site and recreated it again. The same thing happens again. Although I got a little more detailed message for one of my early failed git push commands:

Uploading LFS objects:  37% (1094/2968), 220 MB | 767 KB/s, done.
Repository or object not found: https://[SITE_ID].netlify.com/.netlify/large-media/verify
Check that it exists and that you have proper access to it
Repository or object not found: https://[SITE_ID].netlify.com/.netlify/large-media/verify
Check that it exists and that you have proper access to it
Post https://[SITE_ID].netlify.com/.netlify/large-media/verify: stream error: stream ID 17; NO_ERROR
Repository or object not found: https://[SITE_ID].netlify.com/.netlify/large-media/verify
Check that it exists and that you have proper access to it
Post https://[SITE_ID].netlify.com/.netlify/large-media/verify: http2: server sent GOAWAY and closed the connection; LastStreamID=17, ErrCode=ENHANCE_YOUR_CALM, debug=""
Repository or object not found: https://[SITE_ID].netlify.com/.netlify/large-media/verify
Check that it exists and that you have proper access to it

batch response: Expected json type, got: "text/plain; charset=utf-8"
batch response: Expected json type, got: "text/plain; charset=utf-8"
...
batch response: Expected json type, got: "text/plain; charset=utf-8"
error: failed to push some refs to 'git@github.com:[ORG]/[REPO].git'

I was able to push all my lfs tracked files to Netlify Large Media servers by splitting my big commit into smaller ones with much lower payloads.

I would still get the same kind of errors but only once per single commit push.

@saw-mon_and_natalie, thank you for sharing this detailed explanation and solution.

I’m asking our developers about this rate limit issue and we’ll update this topic again as soon as we have more information about this.

1 Like

@saw-mon_and_natalie, would you please send us that site id? We’ll be able to research the rate limit with that information.

It is safe to post the site ids publicly.

If you prefer to send that privately, please feel free to private message (PM) any of us on the Netlify support team with that as well (me for example). PMs are enabled for your community login. I double checked to be sure.

1 Like

@luke, I just DMed you my site ids.

Thank you for that information, @saw-mon_and_natalie. I’ve shared the site ids with our Netlify Large Media developers and I’ll follow-up here when we know more.

1 Like