[Support Guide] How do I access private repositories in the build environment?

Last reviewed by Netlify Support on December 2022

The link from your repository to our service only authenticates us against the “main” repo that you first configured, and won’t automatically allow access to submodules even if they also belong to you and you’ve granted Netlify access to those repositories.

In case your submodule is public, you can change the URI schema for it to start with https://, and it should work without further adjustment (so, in .gitmodules, change git@github.com… to https://github.com/… )

If you have a private submodule, you’ll need to grant permissions for our build system to access your private submodule. Since we can link only ONE deploy key to each site, and need to have a separate key for each site, you’ll need to do the following:

  1. Generate your deploy key (a site-specific SSH key) following these instructions

  2. Assign your deploy key as a read-only deploy key in your repository settings at GitHub, which can be found here: https://github.com/<your github account>/<your repository name>/settings/keys.

  3. Once it’s in place, when we check out the first repo to build, we’ll fetch the submodule with this deploy key and this will work!

  4. You can use this workflow with multiple private submodules attached to the same repository, but that does take some extra work since GitHub does not allow Deploy Keys to be used in multiple places. You’ll have to instead add them to a user (perhaps you? or maybe a robot-account that you create on GitHub) that has access to all repos.

If you want to access private repositories not configured as a submodule, it can be a bit of a challenge in the build environment! We do have git and ssh in the build-image, so the tools are there. What’s missing is any authentication credentials at all - we don’t even have permission to fetch the same repo we are building from within the build environment. So, you’ll need to supply whatever you need to access that repository somehow.

There are three ways we know of to do it (and probably a thousand that we do not, feel free to tell us about different ways to do this). The best way to do this requires support of your package manager and Git provider, so it may not be an option depending on your toolchain. All of these options include storing access tokens in your repository or accessing them via secrets stored there.

If you think that could work, there are a couple of ways to get that info into our build environment:

  1. If your dependency manager (this works with npm at least) supports it, and your package.json is securely stored, you can add that dependency using a GitHub or GitLab access token like this: git+https://<github_token>:x-oauth-basic@github.com/<user>/<repo>.git (GitLab also documents this). If you use BitBucket, you can to use an app password in the same way. Note that you can instead choose to not save the token to git, but use it directly from the command line during build if you keep that dependency out of your main package.json and instead set an environment variable like $GIT_TOKEN in the Build & Deploy settings for a site, and using a step in your build process like: npm install git+https://${GIT_TOKEN}:x-oauth-basic@github.com/user/repo.git
  2. Simplest, but not recommended, is to have the ssh private key as part of your repository. Then you can just use it natively (eg if you have /.ssh/id_dsa and it is password-less, ssh will just use it automatically - when used with or without git). This isn’t recommended, because you may have a lot of people with access to your repository that can now access the private dependency!
  3. Instead, you can add some access token to the “Build Environment” section of the settings for your site. This restricts access to only those who can see your Netlify account settings, rather than all who can see your repository/source code. This could be the contents of the SSH key (remember, the ssh private key is multi-line text and formatting-sensitive). It might be easier to pull down the key from a remote server using curl:

curl -dump https://username:${PASSWORD}@hostname/sshkey > ~/.ssh/id_dsa

To use this last trick successfully, you should realize that your build command can be anything you want that can run on a Linux host. See this community post about debugging your site and this blog post for more info about how to tell what is available and test things out locally.

One of my teammates gave a better write-up of how to use an SSH key yourself in this article that might also be useful.

Finally, you can create your own deploy key for a site or replace the existing one, but it is a bit complicated. The workflow described in this article can be used for existing sites as well: [Support Guide] Linking a repository via API

5 Likes