[Support Guide] Using private NPM modules on Netlify

I’m realizing now that the build is not even getting to the user build script – it’s failing when installing yarn:

10:14:17 AM: Build ready to start
10:14:20 AM: build-image version: 9e0f207a27642d0115b1ca97cd5e8cebbe492f63
10:14:20 AM: build-image tag: v3.3.2
10:14:20 AM: buildbot version: 75cd99f62ada9e21edea53208e8baf0eab85a045
10:14:20 AM: Fetching cached dependencies
10:14:21 AM: Starting to download cache of 255.1KB
10:14:21 AM: Finished downloading cache in 104.993978ms
10:14:21 AM: Starting to extract cache
10:14:21 AM: Failed to fetch cache, continuing with build
10:14:21 AM: Starting to prepare the repo for build
10:14:21 AM: No cached dependencies found. Cloning fresh repo
10:14:21 AM: git clone git@gitlab.com:[username]/[repo]
10:14:23 AM: Preparing Git Reference refs/heads/master
10:14:24 AM: Starting build script
10:14:24 AM: Installing dependencies
10:14:26 AM: Downloading and installing node v8.16.0...
10:14:26 AM: Downloading https://nodejs.org/dist/v8.16.0/node-v8.16.0-linux-x64.tar.xz...
10:14:26 AM: 
#
10:14:26 AM:   1.7%
10:14:26 AM: 
#####################
10:14:26 AM:  29.9%
10:14:26 AM: 
################
10:14:26 AM: #################################################         91.0%
10:14:26 AM: 
########################################################################
10:14:26 AM: 100.0%
10:14:27 AM: Computing checksum with sha256sum
10:14:27 AM: Checksums matched!
10:14:29 AM: Now using node v8.16.0 (npm v6.4.1)
10:14:29 AM: Attempting ruby version 2.3.6, read from environment
10:14:30 AM: 
10:14:30 AM: ** WARNING **
10:14:30 AM: Using custom ruby version 2.3.6, this will slow down the build.
10:14:30 AM: To ensure fast builds, set the RUBY_VERSION environment variable, or .ruby-version file, to an included ruby version.
10:14:30 AM: Included versions: 2.6.2
10:14:30 AM: 
10:14:30 AM: Required ruby-2.3.6 is not installed - installing.
10:14:31 AM: Searching for binary rubies, this might take some time.
10:14:31 AM: Found remote file https://rvm_io.global.ssl.fastly.net/binaries/ubuntu/16.04/x86_64/ruby-2.3.6.tar.bz2
10:14:31 AM: Checking requirements for ubuntu.
10:14:32 AM: Requirements installation successful.
10:14:32 AM: ruby-2.3.6 - #configure
10:14:32 AM: ruby-2.3.6 - #download
10:14:33 AM: ruby-2.3.6 - #validate archive
10:14:41 AM: ruby-2.3.6 - #extract
10:14:45 AM: ruby-2.3.6 - #validate binary
10:14:46 AM: ruby-2.3.6 - #setup
10:14:47 AM: ruby-2.3.6 - #gemset created /opt/buildhome/.rvm/gems/ruby-2.3.6@global
10:14:47 AM: ruby-2.3.6 - #importing gemset /opt/buildhome/.rvm/gemsets/global.gems
10:14:49 AM: ............................................................
10:14:49 AM: ruby-2.3.6 - #generating global wrappers
10:14:49 AM: .......
10:14:49 AM: ruby-2.3.6 - #gemset created /opt/buildhome/.rvm/gems/ruby-2.3.6
10:14:49 AM: ruby-2.3.6 - #importing gemsetfile /opt/buildhome/.rvm/gemsets/default.gems evaluated to empty gem list
10:14:49 AM: ruby-2.3.6 - #generating default wrappers
10:14:49 AM: .......
10:14:50 AM: Using /opt/buildhome/.rvm/gems/ruby-2.3.6
10:14:50 AM: Using ruby version 2.3.6
10:14:50 AM: Using PHP version 5.6
10:14:50 AM: Started restoring cached node modules
10:14:50 AM: Finished restoring cached node modules
10:14:50 AM: Started restoring cached yarn cache
10:14:50 AM: Finished restoring cached yarn cache
10:14:50 AM: Installing yarn at version 1.3.2
10:14:50 AM: Installing Yarn!
10:14:50 AM: > Downloading tarball...
10:14:50 AM: [1/2]: https://yarnpkg.com/downloads/1.3.2/yar
10:14:50 AM: n-v1.3.2.tar.gz --> /tmp/yarn.tar.gz.3XCWlG4Gdu
10:14:50 AM:   % Total    % Received % Xferd  Average
10:14:50 AM: Speed   Time    Time     Time  Current
10:14:50 AM:                       Dload  Upload   T
10:14:50 AM: otal   Spent    Left  Speed
10:14:50 AM: 
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:-
10:14:50 AM: -:--     0
10:14:51 AM: 
100    91  100    91    0     0    320      0 --:--:-- --:--:-- --:--:--
10:14:51 AM: 321
10:14:51 AM: 
100   608    0   608    0     0    955
10:14:51 AM:   0 --:--:-- --:--:-- --:--:--   955
10:14:51 AM: 
  0  865k    0     0    0     0      0      0 --:--:-- --:
10:14:51 AM: --:-- --:--:--     0
10:14:51 AM: 
100  865k  100  865k    0     0   796k      0  0:00:01  0:00:01
10:14:51 AM: --:--:-- 5549k
10:14:51 AM: [2/2]: https://yarnpkg
10:14:51 AM: .com/downloads/1.3.2/yarn-v1.3.2.tar.gz.asc --> /tmp/yarn.tar.gz.3XCWlG4Gdu.asc
10:14:51 AM: 
100    95  100    95    0     0   2247      0 --:--:-- --:--
10:14:51 AM: :-- --:--:--  2247
10:14:52 AM: 
100   612    0   612    0     0   3991      0 --:--:-- --:--:
10:14:52 AM: -- --:--:--  3991
10:14:52 AM: 
100  1027  100  1027    0     0   5327      0 --:--:-- --:
10:14:52 AM: --:-- --:--:--  5327
10:14:52 AM: > Verifying integrity...
10:14:52 AM: gpg: Signature made Thu 02 Nov 2017 04:44:10 PM UTC using RSA key ID FD2497F5
10:14:52 AM: gpg: Good signature from "Yarn Packaging <yarn@dan.cx>"
10:14:52 AM: gpg: Note: This key has expired!
10:14:52 AM: Primary key fingerprint: 72EC F46A 56B4 AD39 C907  BBB7 1646 B01B 86E5 0310
10:14:52 AM:      Subkey fingerprint: 6A01 0C51 6600 6599 AA17  F081 46C2 130D
10:14:52 AM:  FD24 97F5
10:14:52 AM: > GPG signature looks good
10:14:52 AM: > Extracting to ~/.yarn...
10:14:52 AM: > Adding to $PATH...
10:14:52 AM: > We've added the following to your /opt/buildhome/.profile
10:14:52 AM: > If this isn't the profile of your current shell then please add the following to your correct profile:
10:14:52 AM: export PATH="$HOME/.yarn/bin:$HOME/.config/yarn/global/node_modules/.bin:$PATH"
10:14:52 AM: 
10:14:52 AM: > Successfully installed Yarn 1.3.2! Please open another terminal where the `yarn` command will now be available.
10:14:53 AM: Installing NPM modules using Yarn version 1.3.2
10:14:53 AM: yarn install v1.3.2
10:14:54 AM: [1/4] Resolving packages...
10:14:54 AM: [2/4] Fetching packages...
10:14:56 AM: error An unexpected error occurred: "https://node.bit.dev/ajsharp.vinvin.booking-time-field/-/ajsharp.vinvin.booking-time-field-1.0.0.tgz: Request failed \"401 Unauthorized\"".
10:14:56 AM: info If you think this is a bug, please open a bug report with the information provided in "/opt/build/repo/booking-plugin/yarn-error.log".
10:14:56 AM: info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
10:15:07 AM: Error during Yarn install
10:15:07 AM: failed during stage 'building site': Build script returned non-zero exit code: 1
10:15:07 AM: Error running command: Build script returned non-zero exit code: 1
10:15:08 AM: Shutting down logging, 2 messages pending

Also, it’s probably worth mentioning that the failing environment is attempt to install and use yarn version 1.3.2, and one working is using yarn 1.13.0.

UPDATE:

Good news: I was able to resolve this by manually setting the yarn version to 1.13.0 by setting the YARN_VERSION environment variable to 1.13.0.

Not clear if this is a yarn bug or a netlify bug, but private npm registries seem broken with newer versions of yarn.

Hi @fool,

Thank you for your great explanation.
Unfortunately it does not work for my project which contains an Angular (v8) App with AOT compilation.
I created an npmrc file and added my npm token as a netlify environment variable but when deploying the app, it seems like the private packages can not be found which is why the deployment fails:

2:50:10 PM: Build ready to start
2:50:15 PM: build-image version: 9e0f207a27642d0115b1ca97cd5e8cebbe492f63
2:50:15 PM: build-image tag: v3.3.2
2:50:15 PM: buildbot version: ef8d0929ed0baabafd8bbb7d0b021e1fc24180c0
2:50:15 PM: Fetching cached dependencies
2:50:15 PM: Starting to download cache of 162.5MB
2:50:16 PM: Finished downloading cache in 1.247109651s
2:50:16 PM: Starting to extract cache
2:50:24 PM: Finished extracting cache in 7.700806118s
2:50:24 PM: Finished fetching cache in 8.991370823s
2:50:24 PM: Starting to prepare the repo for build
2:50:25 PM: Preparing Git Reference refs/heads/master
2:50:25 PM: Starting build script
2:50:25 PM: Installing dependencies
2:50:26 PM: Started restoring cached node version
2:50:28 PM: Finished restoring cached node version
2:50:29 PM: v10.16.0 is already installed.
2:50:30 PM: Now using node v10.16.0 (npm v6.9.0)
2:50:30 PM: Attempting ruby version 2.6.2, read from environment
2:50:31 PM: Using ruby version 2.6.2
2:50:32 PM: Using PHP version 5.6
2:50:32 PM: Started restoring cached node modules
2:50:32 PM: Finished restoring cached node modules
2:50:32 PM: Installing NPM modules using NPM version 6.9.0
2:51:20 PM: > core-js@2.6.9 postinstall /opt/build/repo/node_modules/babel-runtime/node_modules/core-js
2:51:20 PM: > node scripts/postinstall || echo "ignore"
2:51:20 PM: Thank you for using core-js ( https://github.com/zloirock/core-js ) for polyfilling JavaScript standard library!
2:51:20 PM: The project needs your help! Please consider supporting of core-js on Open Collective or Patreon: 
2:51:20 PM: > https://opencollective.com/core-js 
2:51:20 PM: > https://www.patreon.com/zloirock 
2:51:20 PM: Also, the author of core-js ( https://github.com/zloirock ) is looking for a good job -)
2:51:20 PM: > core-js@2.6.9 postinstall /opt/build/repo/node_modules/karma/node_modules/core-js
2:51:20 PM: > node scripts/postinstall || echo "ignore"
2:51:20 PM: Thank you for using core-js ( https://github.com/zloirock/core-js ) for polyfilling JavaScript standard library!
2:51:20 PM: The project needs your help! Please consider supporting of core-js on Open Collective or Patreon: 
2:51:20 PM: > https://opencollective.com/core-js 
2:51:20 PM: > https://www.patreon.com/zloirock 
2:51:20 PM: Also, the author of core-js ( https://github.com/zloirock ) is looking for a good job -)
2:51:20 PM: > @angular/cli@8.0.6 postinstall /opt/build/repo/node_modules/@angular/cli
2:51:20 PM: > node ./bin/postinstall/script.js
2:51:21 PM: > core-js@3.1.4 postinstall /opt/build/repo/node_modules/core-js
2:51:21 PM: > node scripts/postinstall || echo "ignore"
2:51:21 PM: Thank you for using core-js ( https://github.com/zloirock/core-js ) for polyfilling JavaScript standard library!
2:51:21 PM: The project needs your help! Please consider supporting of core-js on Open Collective or Patreon: 
2:51:21 PM: > https://opencollective.com/core-js 
2:51:21 PM: > https://www.patreon.com/zloirock 
2:51:21 PM: Also, the author of core-js ( https://github.com/zloirock ) is looking for a good job -)
2:51:22 PM: npm notice created a lockfile as package-lock.json. You should commit this file.
2:51:22 PM: npm
2:51:22 PM: WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.2.9 (node_modules/fsevents):
2:51:22 PM: npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.9: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
2:51:22 PM: added 101 packages from 65 contributors, removed 171 packages, updated 236 packages, moved 7 packages and audited 19366 packages in 48.976s
2:51:22 PM: found 0 vulnerabilities
2:51:22 PM: NPM modules installed
2:51:22 PM: Started restoring cached go cache
2:51:22 PM: Finished restoring cached go cache
2:51:22 PM: unset GOOS;
2:51:22 PM: unset GOARCH;
2:51:22 PM: export GOROOT='/opt/buildhome/.gimme/versions/go1.12.linux.amd64';
2:51:22 PM: export PATH="/opt/buildhome/.gimme/versions/go1.12.linux.amd64/bin:${PATH}";
2:51:22 PM: go version >&2;
2:51:22 PM: export GIMME_ENV='/opt/buildhome/.gimme/env/go1.12.linux.amd64.env';
2:51:22 PM: go version go1.12 linux/amd64
2:51:22 PM: Installing missing commands
2:51:22 PM: Verify run directory
2:51:22 PM: Executing user command: npm run build-staging
2:51:23 PM: > done-webapp@0.0.0 build-staging /opt/build/repo
2:51:23 PM: > ng build --prod --aot --configuration=staging
2:51:51 PM: Date: 2019-07-02T12:51:51.687Z
2:51:51 PM: Hash: e1fd16cadfe7aff533f4
2:51:51 PM: Time: 23973ms
2:51:51 PM: chunk {0} runtime-es5.741402d1d47331ce975c.js (runtime) 1.41 kB [entry] [rendered]
2:51:51 PM: chunk {1} main-es5.4af9b61479361f268d39.js (main) 128 bytes [initial] [rendered]
2:51:52 PM: failed during stage 'building site': Build script returned non-zero exit code: 1
2:51:51 PM: chunk {2} polyfills-es5.5639a156d6529acb291d.js (polyfills) 68.1 kB [initial] [rendered]
2:51:51 PM: ERROR in src/app/platform/shared/pressure-canvas/pressure-canvas.component.ts(9,33): error TS2307: Cannot find module '@done/sensor-client'.
2:51:51 PM: src/app/platform/shared/pressure-canvas/pressure-canvas.component.ts(10,38): error TS2307: Cannot find module '@done/sensor-client-player'.
2:51:51 PM: src/app/platform/shared/pressure-canvas/pressure-canvas.component.ts(12,46): error TS2307: Cannot find module '@done/sensor-client-edit'.
2:51:51 PM: src/app/platform/shared/pressure-canvas/pressure-canvas.component.ts(13,24): error TS2307: Cannot find module '@done/sensor-client-viewer'.
2:51:51 PM: npm
2:51:51 PM:  ERR! code ELIFECYCLE
2:51:51 PM: npm
2:51:51 PM: ERR! errno 1
2:51:51 PM: npm
2:51:51 PM: ERR!
2:51:51 PM:  done-webapp@0.0.0 build-staging: `ng build --prod --aot --configuration=staging`
2:51:51 PM: npm
2:51:51 PM: ERR! Exit status 1
2:51:51 PM: npm
2:51:51 PM:  ERR!
2:51:51 PM: npm
2:51:51 PM: ERR! Failed at the done-webapp@0.0.0 build-staging script.
2:51:51 PM: npm
2:51:51 PM: ERR! This is probably not a problem with npm. There is likely additional logging output above.
2:51:52 PM: npm
2:51:52 PM:  ERR! A complete log of this run can be found in:
2:51:52 PM: npm ERR!     /opt/buildhome/.npm/_logs/2019-07-02T12_51_51_766Z-debug.log
2:51:52 PM: Skipping functions preparation step: no functions directory set
2:51:52 PM: Caching artifacts
2:51:52 PM: Started saving node modules
2:51:52 PM: Finished saving node modules

The four modules which can not be found are the private repos.
When I start the build on my personal machine, everything works as expected.

Thank you for your help.
Stefan

I don’t see the attempt to install those modules in your logs? They would need to be in package.json, and a failure to install them should produce an error BEFORE your build starts. Perhaps you have them set as devDependencies and you are using NODE_ENV=production or something? Also this workflow is for installing private NPM modules - not code from private repos, and you may have been using the term interchangeably but I wanted to make sure you were talking about the same thing as me :slight_smile:

1 Like

Hi @fool

Thanks for your response.
This is exactly what makes me wondering as well.
I do have the packages set in the dependencies and not in the devDependencies. And yes I’m talking about private npm modules and not private repos :slight_smile:

There is indeed an error if I remove the .npmrc which says that the four private npm modules can not be found on npm (because they’re private). That’s actually the reason why I got here and followed your advice :slight_smile:
So I guess there is an attempt to install those modules but it’s not shown? I’m not sure and stuck a little that’s why I’m reaching out to you.
Another thought that came in my mind is that the build has some issues with the Angular AOT compilation? So maybe it checks for the private modules before they have been installed?

Thanks a lot!

I just disabled the AOT compilation and tried the deployment again. But same error. So that’s not the reason I guess.

Ah, well, we won’t re-install the modules if they are already there in the site’s dependency cache from some previous build, and perhaps they are some of the sort that can be installed quietly and wouldn’t say anything. That seems like it was a wrong guess, sorry!

Next step for debugging would be to try to debug in our build environment run locally so you can look around in the filesystem in the build container. This shows how to do that:

1 Like

It’s probably worth mentioning that adding a .npmrc file to your repository breaks nvm locally as it can’t find the NPM_TOKEN environment variable.

It would be great if netlify had a way to either run a script prior to the npm install or just a way to add some repository authorization.

2 Likes

It works fine, if you define $NPM_TOKEN locally which we presumed would be your use case since you’d want the same access locally, right? Even if you don’t otherwise use it for some reason, you could still set it to allow one build script to rule them all :slight_smile:

We do have an open feature request around “do something before netlify installs deps” to which I’ve added this thread so we can notify you and others who might find it if that becomes a reality!

Usually the NPM_TOKEN or any other token on the local machine is configured on the machine .npmrc (~/.npmrc file).
Writing .npmrc locally in the project overrides the global definition.

The problem with setting tokens globally is that they might be prone to changes, which can take a while to figure out.

A local script before install solves this by creating the .npmrc file dynamically:
echo >>.npmrc

On a side issue - it is customary for CI machines to have an environment variable CI set to true, so actions can be executed based on CI. e.g. in the script above.

This is a big issue for us.

I understand you are concerned about two things, and while I can’t offer any relief today around npmrc except making sure that the person working on the retooling of our build environment sees this thread, there are at least many environment variables you can key off of to see that you are in the netlify CI:

Hey! I’m trying to use a package from github package registry. I’ve followed the instructions in this post, but I’m getting this error:

7:48:46 PM: ERR! code
7:48:46 PM:  E401
7:48:46 PM: npm
7:48:46 PM:  ERR! Unable to authenticate, need: Basic realm="GitHub Package Registry"

I’ve made sure that my token has read and write permissions for packages and also repo permissions.

My .npmrc looks like this:

//npm.pkg.github.com:_authToken=${GITHUB_PACKAGES_TOKEN}
@guayom:registry=https://npm.pkg.github.com

And GITHUB_PACKAGES_TOKEN is defined in my env vars

Actually, the repo is public. I just created it for learning purposes

I’ve been trying to publish for a couple hours now, but no luck.
Any ideas?

It seems that the token doesn’t have enough permissions: Basic realm="GitHub Package Registry". Can you share the netlify site that you are trying to deploy that repo on?

Yeah, it is this one: naughty-kare-e205e8
It hasn’t been deployed yet

I can get this working with the NPM_TOKEN env variable and the .npmrc file, but it’s not ideal, and here’s why:

Sometimes I don’t want to create a .npmrc file and push it to git - in this case I’m out of options.
Now.sh implemented this by also exposing a NPM_RC env variable, where you can put the whole contents of the file. This is nice because I don’t want devs to have .npmrc in their repo when they check out the repo - I just want the build server to have a special npm configuration via the NPM_RC variable.

Any chance we could see this implemented? Thanks a lot!

1 Like

You can certainly create that workflow today:

  • wire things so that our automatic installation of things doesn’t happen (maybe add an NPM_FLAGS of --dry-run ?)
  • have your build check for a variable you set only at netlify (or you can just check for $NETLIFY or other variables we set at build time: Build environment variables | Netlify Docs
  • use that to “gate” your build on first constructing .npmrc correctly based on variables in the account, or use the default settings instead if it is a local build.

But to your point of making the build more configurable, that is the goal of our new build plugins system: Netlify Build Plugins: add powerful capabilities to every build where you can more easily make your dreams a reality and I think that is the best path to get things working to match your precise needs, since we won’t make very big changes to our existing production (soon to be legacy) less configurable build system.

1 Like

Thanks for the tips. I was trying to create something along the same lines - to generate the correct .npmrc file (only when a custom $NETLIFY variable was set) - but was doing that in the preinstall npm script, which was already too late.

Will try your proposal and come back with feedback - thank you.
Also, thanks for the heads-up with the new build plugins, already requested an invite for that.

1 Like

When i’m trying npm install --dry-run in order to get packages from a private npm registry, the command will still fail - so dry run is not a solution (when I try --dry-run locally it succeeds when I’m logged into the private registry, but fails when I’m not logged in - so it will obviously fail in the netlify build process as well).

1 Like

Darn, we thought this would be the solution but it indeed doesn’t seem to work. My guess is the npm process won’t pick up any more changes to the configuration once it has started running.

Our team figured out a *cough* slightly hacky *cough* way to get this to work with the npm preinstall script.

Create a preinstall script (for example at ./scripts/netlify-preinstall.js):

const fs = require('fs')
const { spawnSync } = require('child_process')

// Netlify does not support Github Packages (or other private package registries besides npm), options are:
//   - Commit .npmrc to repo - However, now we have a secret token inside our repo
//   - Environment variable in .npmrc - However, this requires all developer machines to have the same environment variable configured
//   - Get creative with the preinstall script... :)

// Only run this script on Netlify
if (process.env.NETLIFY === 'true') { // this is a default Netlify environment variable
	// Check if .npmrc already exists, if it does then do nothing (otherwise we create an infinite yarn loop)
	if (!fs.existsSync('.npmrc')) {
		// Create .npmrc
		fs.writeFileSync('.npmrc', `//npm.pkg.github.com/:_authToken=${process.env.GITHUB_TOKEN}\n@oliverit:registry=https://npm.pkg.github.com/\n`)
		fs.chmodSync('.npmrc', 0o600)
		// Run yarn again, because the yarn process which is executing
		// this script won't pick up the .npmrc file we just created.
		// The original yarn process will continue after this second yarn process finishes,
		// and when it does it will report "success Already up-to-date."
		spawnSync('yarn', { stdio: 'inherit' })
	}
}

Configure npm/yarn to run this script before installing dependencies by defining the preinstall script inside the package.json file:

  "scripts": {
    "preinstall": "node ./scripts/netlify-preinstall.js"
  },

Don’t forget to add the GITHUB_TOKEN variable in your Netlify team or site settings.

1 Like