[Common Issue] Using private NPM modules on Netlify

answered
needs_documentation
#1

You want to build your site on Netlify, but you have some private npm modules served from npmjs.org. So how can you access them during build?

The most straightforward way to do this is two steps:

  1. Find your token inside the ~/.npmrc file in your home folder and set it as an NPM_TOKEN environment variable through Netlify’s admin UI. This will be on the Build & Deploy settings page of your site configuration, in the section called “Build Environment Variables”:

  1. Create a /.npmrc file for your project like this (and commit it to Git):

//registry.npmjs.org/:_authToken=${NPM_TOKEN}

That should let us automatically pull the module from npm using your token, and your token does NOT need to be committed to your repo. If you’d rather have it there than in Netlify for some reason, that’s also functional :slight_smile: If you store it at Netlify the only people who can see it are people whom you’ve invited to your Netlify account; if you store it in your repo, it’s instead visible to everyone who can see your repo.

Some other related information on the topic can be found in the following threads:

  1. This thread in the npm community talks about how npm uses the environment, which explains why your .env file won’t “just work” in this case.
  2. If you need to set up access to something OTHER than a module to fetch from npm, you might instead find this post in our community to be useful. It describes accessing secondary repositories during build.
4 Likes
#2

Unfortunately, I couldn’t get this to work with a local project .npmrc and an environment variable. It did, however work, if I inline the environment variable into the committed .npmrc file.

I was also trying to get it to work by creating a npmrc on the fly during build, but that also wasn’t working. Would love to figure out a way to get this working without committing creds to source control.

#3

Hmm, what happened when you tried locally? Perhaps there was a typo in my spec. I don’t have a private npm module to test with, but the workflow has been handed out to a few dozen customers in the time I’ve worked here and none have reported trouble with it, so I think once you get it working locally it should work here too :slight_smile: This is the (local) test I’d do ($ is the prompt on my mac, the other commands can be cut and pasted into the SAME shell window as long as you run a BASH-like shell to use as is, with the exception of replacing NPM_TOKEN with your own token):

$ export NPM_TOKEN=00000-000-000-0000000
$ cd project/
$ echo "//registry.npmjs.org/:_authToken=${NPM_TOKEN}" >> .npmrc # don't clobber other existing settings
$ npm i
#4

Apologies on the slow reply – missed the notification.

I don’t think it matters, but I’m actually trying to do this with bit.dev, which works the same way as any private npm registry. Here’s their docs on how to set this up, which is very similar to what you’re recommending here.

Locally, on my dev machine, this all works fine. I’m inlining the auth token in my machine-wide ~/.npmrc and the project local .npmrc, b/c the npm environment variable syntax is weird and breaks some stuff in my local project config.

Currently on netlify i’m running into some weird behavior where two identical environment configurations fail on one and succeed on another. Same codebase (staging/production for the same codebase). On one netlify environment yarn install succeeds, on the other it fails with a 401 for the private repo.

If there was a way to ssh into netlify I’d be able to diagnose the issue better, but afaik this is not possible.

#5

I’m realizing now that the build is not even getting to the user build script – it’s failing when installing yarn:

10:14:17 AM: Build ready to start
10:14:20 AM: build-image version: 9e0f207a27642d0115b1ca97cd5e8cebbe492f63
10:14:20 AM: build-image tag: v3.3.2
10:14:20 AM: buildbot version: 75cd99f62ada9e21edea53208e8baf0eab85a045
10:14:20 AM: Fetching cached dependencies
10:14:21 AM: Starting to download cache of 255.1KB
10:14:21 AM: Finished downloading cache in 104.993978ms
10:14:21 AM: Starting to extract cache
10:14:21 AM: Failed to fetch cache, continuing with build
10:14:21 AM: Starting to prepare the repo for build
10:14:21 AM: No cached dependencies found. Cloning fresh repo
10:14:21 AM: git clone git@gitlab.com:[username]/[repo]
10:14:23 AM: Preparing Git Reference refs/heads/master
10:14:24 AM: Starting build script
10:14:24 AM: Installing dependencies
10:14:26 AM: Downloading and installing node v8.16.0...
10:14:26 AM: Downloading https://nodejs.org/dist/v8.16.0/node-v8.16.0-linux-x64.tar.xz...
10:14:26 AM: 
#
10:14:26 AM:   1.7%
10:14:26 AM: 
#####################
10:14:26 AM:  29.9%
10:14:26 AM: 
################
10:14:26 AM: #################################################         91.0%
10:14:26 AM: 
########################################################################
10:14:26 AM: 100.0%
10:14:27 AM: Computing checksum with sha256sum
10:14:27 AM: Checksums matched!
10:14:29 AM: Now using node v8.16.0 (npm v6.4.1)
10:14:29 AM: Attempting ruby version 2.3.6, read from environment
10:14:30 AM: 
10:14:30 AM: ** WARNING **
10:14:30 AM: Using custom ruby version 2.3.6, this will slow down the build.
10:14:30 AM: To ensure fast builds, set the RUBY_VERSION environment variable, or .ruby-version file, to an included ruby version.
10:14:30 AM: Included versions: 2.6.2
10:14:30 AM: 
10:14:30 AM: Required ruby-2.3.6 is not installed - installing.
10:14:31 AM: Searching for binary rubies, this might take some time.
10:14:31 AM: Found remote file https://rvm_io.global.ssl.fastly.net/binaries/ubuntu/16.04/x86_64/ruby-2.3.6.tar.bz2
10:14:31 AM: Checking requirements for ubuntu.
10:14:32 AM: Requirements installation successful.
10:14:32 AM: ruby-2.3.6 - #configure
10:14:32 AM: ruby-2.3.6 - #download
10:14:33 AM: ruby-2.3.6 - #validate archive
10:14:41 AM: ruby-2.3.6 - #extract
10:14:45 AM: ruby-2.3.6 - #validate binary
10:14:46 AM: ruby-2.3.6 - #setup
10:14:47 AM: ruby-2.3.6 - #gemset created /opt/buildhome/.rvm/gems/ruby-2.3.6@global
10:14:47 AM: ruby-2.3.6 - #importing gemset /opt/buildhome/.rvm/gemsets/global.gems
10:14:49 AM: ............................................................
10:14:49 AM: ruby-2.3.6 - #generating global wrappers
10:14:49 AM: .......
10:14:49 AM: ruby-2.3.6 - #gemset created /opt/buildhome/.rvm/gems/ruby-2.3.6
10:14:49 AM: ruby-2.3.6 - #importing gemsetfile /opt/buildhome/.rvm/gemsets/default.gems evaluated to empty gem list
10:14:49 AM: ruby-2.3.6 - #generating default wrappers
10:14:49 AM: .......
10:14:50 AM: Using /opt/buildhome/.rvm/gems/ruby-2.3.6
10:14:50 AM: Using ruby version 2.3.6
10:14:50 AM: Using PHP version 5.6
10:14:50 AM: Started restoring cached node modules
10:14:50 AM: Finished restoring cached node modules
10:14:50 AM: Started restoring cached yarn cache
10:14:50 AM: Finished restoring cached yarn cache
10:14:50 AM: Installing yarn at version 1.3.2
10:14:50 AM: Installing Yarn!
10:14:50 AM: > Downloading tarball...
10:14:50 AM: [1/2]: https://yarnpkg.com/downloads/1.3.2/yar
10:14:50 AM: n-v1.3.2.tar.gz --> /tmp/yarn.tar.gz.3XCWlG4Gdu
10:14:50 AM:   % Total    % Received % Xferd  Average
10:14:50 AM: Speed   Time    Time     Time  Current
10:14:50 AM:                       Dload  Upload   T
10:14:50 AM: otal   Spent    Left  Speed
10:14:50 AM: 
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:-
10:14:50 AM: -:--     0
10:14:51 AM: 
100    91  100    91    0     0    320      0 --:--:-- --:--:-- --:--:--
10:14:51 AM: 321
10:14:51 AM: 
100   608    0   608    0     0    955
10:14:51 AM:   0 --:--:-- --:--:-- --:--:--   955
10:14:51 AM: 
  0  865k    0     0    0     0      0      0 --:--:-- --:
10:14:51 AM: --:-- --:--:--     0
10:14:51 AM: 
100  865k  100  865k    0     0   796k      0  0:00:01  0:00:01
10:14:51 AM: --:--:-- 5549k
10:14:51 AM: [2/2]: https://yarnpkg
10:14:51 AM: .com/downloads/1.3.2/yarn-v1.3.2.tar.gz.asc --> /tmp/yarn.tar.gz.3XCWlG4Gdu.asc
10:14:51 AM: 
100    95  100    95    0     0   2247      0 --:--:-- --:--
10:14:51 AM: :-- --:--:--  2247
10:14:52 AM: 
100   612    0   612    0     0   3991      0 --:--:-- --:--:
10:14:52 AM: -- --:--:--  3991
10:14:52 AM: 
100  1027  100  1027    0     0   5327      0 --:--:-- --:
10:14:52 AM: --:-- --:--:--  5327
10:14:52 AM: > Verifying integrity...
10:14:52 AM: gpg: Signature made Thu 02 Nov 2017 04:44:10 PM UTC using RSA key ID FD2497F5
10:14:52 AM: gpg: Good signature from "Yarn Packaging <yarn@dan.cx>"
10:14:52 AM: gpg: Note: This key has expired!
10:14:52 AM: Primary key fingerprint: 72EC F46A 56B4 AD39 C907  BBB7 1646 B01B 86E5 0310
10:14:52 AM:      Subkey fingerprint: 6A01 0C51 6600 6599 AA17  F081 46C2 130D
10:14:52 AM:  FD24 97F5
10:14:52 AM: > GPG signature looks good
10:14:52 AM: > Extracting to ~/.yarn...
10:14:52 AM: > Adding to $PATH...
10:14:52 AM: > We've added the following to your /opt/buildhome/.profile
10:14:52 AM: > If this isn't the profile of your current shell then please add the following to your correct profile:
10:14:52 AM: export PATH="$HOME/.yarn/bin:$HOME/.config/yarn/global/node_modules/.bin:$PATH"
10:14:52 AM: 
10:14:52 AM: > Successfully installed Yarn 1.3.2! Please open another terminal where the `yarn` command will now be available.
10:14:53 AM: Installing NPM modules using Yarn version 1.3.2
10:14:53 AM: yarn install v1.3.2
10:14:54 AM: [1/4] Resolving packages...
10:14:54 AM: [2/4] Fetching packages...
10:14:56 AM: error An unexpected error occurred: "https://node.bit.dev/ajsharp.vinvin.booking-time-field/-/ajsharp.vinvin.booking-time-field-1.0.0.tgz: Request failed \"401 Unauthorized\"".
10:14:56 AM: info If you think this is a bug, please open a bug report with the information provided in "/opt/build/repo/booking-plugin/yarn-error.log".
10:14:56 AM: info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
10:15:07 AM: Error during Yarn install
10:15:07 AM: failed during stage 'building site': Build script returned non-zero exit code: 1
10:15:07 AM: Error running command: Build script returned non-zero exit code: 1
10:15:08 AM: Shutting down logging, 2 messages pending
#6

Also, it’s probably worth mentioning that the failing environment is attempt to install and use yarn version 1.3.2, and one working is using yarn 1.13.0.

UPDATE:

Good news: I was able to resolve this by manually setting the yarn version to 1.13.0 by setting the YARN_VERSION environment variable to 1.13.0.

Not clear if this is a yarn bug or a netlify bug, but private npm registries seem broken with newer versions of yarn.