Wrong link generation to gitlab when using folders subfolders at collection

Support
1

Self hosted Gitlab, gotrue and git-gateway
404 error when trying request a file (tree working)

as shows logs from git-gateway
path=/gitlab/repository/files/XXX/2019/1.json/raw
status=404

but when changing config.yml
folder to /
everything works ok.

browser console shows right format with %2F
/.netlify/git/gitlab/repository/files/XXX%2F**2019%2F1.json/raw?ref=develop&ts=1577203026756

but

log from git-gateway changed after /files/ %2F to /
msg="Proxying to GitLab: https://git.xx/api/v4/projects/prj%2Fsource/repository/files/XXX/2019/1.json/raw?ref=develop&ts=1577203026767

Also same request from gitlab log
"status":404,"method":"GET","path":"/api/v4/projects/prj%2Fsource/repository/files/XXX/2019/1.json/raw"

If i made direct request to gitlab api with 1st variant - i got correct answer.

https://git.xx/api/v4/projects/prj%2Fsource/repository/XXX%2F2019%2F1.json/raw?ref=develop&ts=1577183377446

Someone know fix?
tx

2

Hmm, quite odd. Could you tell us the real URL that is being fetched rather than “git.xx” which is not your real server name, so we can look in our internal logs to see if we have more details?

4

ok. maybe ill explain more clearly
i use selfhosted gotrue
with config

GOTRUE_SITE_URL=https://test.site
GOTRUE_API_HOST=0.0.0.0
PORT=8098
GOTRUE_DB_DRIVER=mysql
DATABASE_URL=user:user@tcp(10.201.0.136:3306)/gotrue?parseTime=true&multiStatements=true
GOTRUE_LOG_LEVEL=DEBUG
GOTRUE_JWT_SECRET=XXX
GOTRUE_JWT_EXP=3600
GOTRUE_JWT_AUD=""

also selfhosted git-gateway

GITGATEWAY_JWT_SECRET=XXX
GITGATEWAY_DB_DRIVER=sqlite3
DATABASE_URL=gorm.db
GITGATEWAY_API_HOST=0.0.0.0
PORT=9999
GITGATEWAY_GITLAB_ENDPOINT=https://git.site/api/v4
GITGATEWAY_GITLAB_ACCESS_TOKEN_TYPE=personal_access
GITGATEWAY_GITLAB_ACCESS_TOKEN=axxxxxb
GITGATEWAY_GITLAB_REPO=opf/opfsource
GITGATEWAY_API_ROOT=https://git.site/api/v4

Selfhosted gitlab
url https://git.site

All services works in internal network, and not reachable from the world

configuration for netlify-gotrue
    backend:
      name: git-gateway
      site_domain: https://test.site
      api_root: https://git.site/api/v4
      base_url: https://git.site
      auth_endpoint: oauth/authorize
      branch: develop
    collections:
      - label: '2019'
        name: 2019"
        folder: "/"

this confihuration works good, but all created files in the root of gitlab repo

log from git-gateway when reauestiong file

msg="Proxying to GitLab: https://git.site/api/v4/projects/opf%2opfsource/repository/files/1-1.json/raw?ref=develop&ts=1577359501242
...
msg="request completed" component=api duration=30941632 gitlab_lb= gitlab_ratelimit_remaining= gitlab_request_id= method=GET path=/gitlab/repository/files/1-1.json/raw referer="https://test.site/admin/" remote_addr="10.255.0.2:35690" request_id=b172e919-17d1-4b91-b01b-d24df6a8f879 status=200

and log from gitlab

"GET /api/v4/projects/opf%2Fopfsource/repository/files/1-1.json/raw?ref=develop&ts=1577359501242 HTTP/1.1"

Now, if i change folder for collection fo example to
collections:
folder: /budget_process

i got this logs
git-gateway

msg="Proxying to GitLab: https://git.test/api/v4/projects/opf%2Fopfsource/repository/files/budget_process/2018/2019-12-20-15-11-3.json/raw?ref=develop&ts=1577360243538"

as you see after files/
budget_process/2018/
now looks like / and not %2F

but im the browser console this part looks good
:path:/.netlify/git/gitlab/repository/files/budget_process%2F2018%2F2019-12-20-15-11-3.json/raw?ref=develop&ts=1577360243538

and finaly same request from gitlab also after /files without %2F

{"time":"2019-12-26T11:37:24.857Z","severity":"INFO","duration":0.72,"db":0.0,"view":0.72,"status":404,"method":"GET","path":"/api/v4/projects/opf%2Fopfsource/repository/files/budget_process/2018/2019-12-20-15-11-3.json/raw","params":{"ref":"develop","ts":"1577360243538"},"host":"git.site","ip":"10.X.X.X, 10.X.X.X, 10.X.X.X","ua":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36","route":"/api/:version/*path","queue_duration":4.69,"gitaly_calls":0}

and this line
/api/v4/projects/opf%2Fopfsource/repository/files/budget_process/2018/2019-12-20-15-11-3.json/raw
with status 404 in browser

right syntax must be
/api/v4/projects/opf%2Fopfsource/repository/files/budget_process%2F2018%2F2019-12-20-15-11-3.json/raw
last one in browser returns 200 status

as i understand something works incorrect at file
git-gateway/api/gitlab.go
near line 47

r.URL.Opaque = "//" + target.Host + singleJoiningSlash(target.EscapedPath(), gitlabPathRegexp.ReplaceAllString(r.URL.EscapedPath(), "/"))

but im not GO developer and not sure how to fix it.

Problems only when requesting /files/
commits and tree didnt use / and works ok now.

latest version of git-gateway, build with docker from github
latest version of gotrue , same docker from github
netlify-cms@2.10.2

Thanks for help

5

ok, finaly…
it works correct with netlify-cms@2.9.8-beta.5 (2.9.7 also)
and from 2.10.0-beta.0, as described previously

update
i was too fast. problem still the same :frowning:

6

Hi @heopz is the problem that the URL encoding for %2F isn’t being maintained by git-gateway? I recommend opening an issue on the repo for git-gateway: Issues · netlify/git-gateway · GitHub If i’m not understanding, please feel free to correct me. However in either case it makes sense to get an issue opened with relevant info so our devs can take a look. If you feel comfortable you can put in a PR yourself as well.

7

Thanks @futuregerald, it was opened 1 day before your post by another guy

9

Hey @heopz, we are still looking into this, but at the moment we don’t have a timeline to fix.

Once we do. we’ll update this thread and let you know, but i also suggest subscribing to that issue in GitHub, so you’ll get notified if someone submits a PR to fix. If you would like to submit your own that would also be very welcome!