Also, it’s probably worth mentioning that the failing environment is attempt to install and use yarn version 1.3.2, and one working is using yarn 1.13.0.
UPDATE:
Good news: I was able to resolve this by manually setting the yarn version to 1.13.0 by setting the YARN_VERSION environment variable to 1.13.0.
Not clear if this is a yarn bug or a netlify bug, but private npm registries seem broken with newer versions of yarn.
Thank you for your great explanation.
Unfortunately it does not work for my project which contains an Angular (v8) App with AOT compilation.
I created an npmrc file and added my npm token as a netlify environment variable but when deploying the app, it seems like the private packages can not be found which is why the deployment fails:
2:50:10 PM: Build ready to start
2:50:15 PM: build-image version: 9e0f207a27642d0115b1ca97cd5e8cebbe492f63
2:50:15 PM: build-image tag: v3.3.2
2:50:15 PM: buildbot version: ef8d0929ed0baabafd8bbb7d0b021e1fc24180c0
2:50:15 PM: Fetching cached dependencies
2:50:15 PM: Starting to download cache of 162.5MB
2:50:16 PM: Finished downloading cache in 1.247109651s
2:50:16 PM: Starting to extract cache
2:50:24 PM: Finished extracting cache in 7.700806118s
2:50:24 PM: Finished fetching cache in 8.991370823s
2:50:24 PM: Starting to prepare the repo for build
2:50:25 PM: Preparing Git Reference refs/heads/master
2:50:25 PM: Starting build script
2:50:25 PM: Installing dependencies
2:50:26 PM: Started restoring cached node version
2:50:28 PM: Finished restoring cached node version
2:50:29 PM: v10.16.0 is already installed.
2:50:30 PM: Now using node v10.16.0 (npm v6.9.0)
2:50:30 PM: Attempting ruby version 2.6.2, read from environment
2:50:31 PM: Using ruby version 2.6.2
2:50:32 PM: Using PHP version 5.6
2:50:32 PM: Started restoring cached node modules
2:50:32 PM: Finished restoring cached node modules
2:50:32 PM: Installing NPM modules using NPM version 6.9.0
2:51:20 PM: > core-js@2.6.9 postinstall /opt/build/repo/node_modules/babel-runtime/node_modules/core-js
2:51:20 PM: > node scripts/postinstall || echo "ignore"
2:51:20 PM: Thank you for using core-js ( https://github.com/zloirock/core-js ) for polyfilling JavaScript standard library!
2:51:20 PM: The project needs your help! Please consider supporting of core-js on Open Collective or Patreon:
2:51:20 PM: > https://opencollective.com/core-js
2:51:20 PM: > https://www.patreon.com/zloirock
2:51:20 PM: Also, the author of core-js ( https://github.com/zloirock ) is looking for a good job -)
2:51:20 PM: > core-js@2.6.9 postinstall /opt/build/repo/node_modules/karma/node_modules/core-js
2:51:20 PM: > node scripts/postinstall || echo "ignore"
2:51:20 PM: Thank you for using core-js ( https://github.com/zloirock/core-js ) for polyfilling JavaScript standard library!
2:51:20 PM: The project needs your help! Please consider supporting of core-js on Open Collective or Patreon:
2:51:20 PM: > https://opencollective.com/core-js
2:51:20 PM: > https://www.patreon.com/zloirock
2:51:20 PM: Also, the author of core-js ( https://github.com/zloirock ) is looking for a good job -)
2:51:20 PM: > @angular/cli@8.0.6 postinstall /opt/build/repo/node_modules/@angular/cli
2:51:20 PM: > node ./bin/postinstall/script.js
2:51:21 PM: > core-js@3.1.4 postinstall /opt/build/repo/node_modules/core-js
2:51:21 PM: > node scripts/postinstall || echo "ignore"
2:51:21 PM: Thank you for using core-js ( https://github.com/zloirock/core-js ) for polyfilling JavaScript standard library!
2:51:21 PM: The project needs your help! Please consider supporting of core-js on Open Collective or Patreon:
2:51:21 PM: > https://opencollective.com/core-js
2:51:21 PM: > https://www.patreon.com/zloirock
2:51:21 PM: Also, the author of core-js ( https://github.com/zloirock ) is looking for a good job -)
2:51:22 PM: npm notice created a lockfile as package-lock.json. You should commit this file.
2:51:22 PM: npm
2:51:22 PM: WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.2.9 (node_modules/fsevents):
2:51:22 PM: npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.9: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
2:51:22 PM: added 101 packages from 65 contributors, removed 171 packages, updated 236 packages, moved 7 packages and audited 19366 packages in 48.976s
2:51:22 PM: found 0 vulnerabilities
2:51:22 PM: NPM modules installed
2:51:22 PM: Started restoring cached go cache
2:51:22 PM: Finished restoring cached go cache
2:51:22 PM: unset GOOS;
2:51:22 PM: unset GOARCH;
2:51:22 PM: export GOROOT='/opt/buildhome/.gimme/versions/go1.12.linux.amd64';
2:51:22 PM: export PATH="/opt/buildhome/.gimme/versions/go1.12.linux.amd64/bin:${PATH}";
2:51:22 PM: go version >&2;
2:51:22 PM: export GIMME_ENV='/opt/buildhome/.gimme/env/go1.12.linux.amd64.env';
2:51:22 PM: go version go1.12 linux/amd64
2:51:22 PM: Installing missing commands
2:51:22 PM: Verify run directory
2:51:22 PM: Executing user command: npm run build-staging
2:51:23 PM: > done-webapp@0.0.0 build-staging /opt/build/repo
2:51:23 PM: > ng build --prod --aot --configuration=staging
2:51:51 PM: Date: 2019-07-02T12:51:51.687Z
2:51:51 PM: Hash: e1fd16cadfe7aff533f4
2:51:51 PM: Time: 23973ms
2:51:51 PM: chunk {0} runtime-es5.741402d1d47331ce975c.js (runtime) 1.41 kB [entry] [rendered]
2:51:51 PM: chunk {1} main-es5.4af9b61479361f268d39.js (main) 128 bytes [initial] [rendered]
2:51:52 PM: failed during stage 'building site': Build script returned non-zero exit code: 1
2:51:51 PM: chunk {2} polyfills-es5.5639a156d6529acb291d.js (polyfills) 68.1 kB [initial] [rendered]
2:51:51 PM: ERROR in src/app/platform/shared/pressure-canvas/pressure-canvas.component.ts(9,33): error TS2307: Cannot find module '@done/sensor-client'.
2:51:51 PM: src/app/platform/shared/pressure-canvas/pressure-canvas.component.ts(10,38): error TS2307: Cannot find module '@done/sensor-client-player'.
2:51:51 PM: src/app/platform/shared/pressure-canvas/pressure-canvas.component.ts(12,46): error TS2307: Cannot find module '@done/sensor-client-edit'.
2:51:51 PM: src/app/platform/shared/pressure-canvas/pressure-canvas.component.ts(13,24): error TS2307: Cannot find module '@done/sensor-client-viewer'.
2:51:51 PM: npm
2:51:51 PM: ERR! code ELIFECYCLE
2:51:51 PM: npm
2:51:51 PM: ERR! errno 1
2:51:51 PM: npm
2:51:51 PM: ERR!
2:51:51 PM: done-webapp@0.0.0 build-staging: `ng build --prod --aot --configuration=staging`
2:51:51 PM: npm
2:51:51 PM: ERR! Exit status 1
2:51:51 PM: npm
2:51:51 PM: ERR!
2:51:51 PM: npm
2:51:51 PM: ERR! Failed at the done-webapp@0.0.0 build-staging script.
2:51:51 PM: npm
2:51:51 PM: ERR! This is probably not a problem with npm. There is likely additional logging output above.
2:51:52 PM: npm
2:51:52 PM: ERR! A complete log of this run can be found in:
2:51:52 PM: npm ERR! /opt/buildhome/.npm/_logs/2019-07-02T12_51_51_766Z-debug.log
2:51:52 PM: Skipping functions preparation step: no functions directory set
2:51:52 PM: Caching artifacts
2:51:52 PM: Started saving node modules
2:51:52 PM: Finished saving node modules
The four modules which can not be found are the private repos.
When I start the build on my personal machine, everything works as expected.
I don’t see the attempt to install those modules in your logs? They would need to be in package.json, and a failure to install them should produce an error BEFORE your build starts. Perhaps you have them set as devDependencies and you are using NODE_ENV=production or something? Also this workflow is for installing private NPM modules - not code from private repos, and you may have been using the term interchangeably but I wanted to make sure you were talking about the same thing as me
Thanks for your response.
This is exactly what makes me wondering as well.
I do have the packages set in the dependencies and not in the devDependencies. And yes I’m talking about private npm modules and not private repos
There is indeed an error if I remove the .npmrc which says that the four private npm modules can not be found on npm (because they’re private). That’s actually the reason why I got here and followed your advice
So I guess there is an attempt to install those modules but it’s not shown? I’m not sure and stuck a little that’s why I’m reaching out to you.
Another thought that came in my mind is that the build has some issues with the Angular AOT compilation? So maybe it checks for the private modules before they have been installed?
Ah, well, we won’t re-install the modules if they are already there in the site’s dependency cache from some previous build, and perhaps they are some of the sort that can be installed quietly and wouldn’t say anything. That seems like it was a wrong guess, sorry!
Next step for debugging would be to try to debug in our build environment run locally so you can look around in the filesystem in the build container. This shows how to do that:
It works fine, if you define $NPM_TOKEN locally which we presumed would be your use case since you’d want the same access locally, right? Even if you don’t otherwise use it for some reason, you could still set it to allow one build script to rule them all
We do have an open feature request around “do something before netlify installs deps” to which I’ve added this thread so we can notify you and others who might find it if that becomes a reality!
Usually the NPM_TOKEN or any other token on the local machine is configured on the machine .npmrc (~/.npmrc file).
Writing .npmrc locally in the project overrides the global definition.
The problem with setting tokens globally is that they might be prone to changes, which can take a while to figure out.
A local script before install solves this by creating the .npmrc file dynamically:
echo >>.npmrc
On a side issue - it is customary for CI machines to have an environment variable CI set to true, so actions can be executed based on CI. e.g. in the script above.
I understand you are concerned about two things, and while I can’t offer any relief today around npmrc except making sure that the person working on the retooling of our build environment sees this thread, there are at least many environment variables you can key off of to see that you are in the netlify CI:
It seems that the token doesn’t have enough permissions: Basic realm="GitHub Package Registry". Can you share the netlify site that you are trying to deploy that repo on?
I can get this working with the NPM_TOKEN env variable and the .npmrc file, but it’s not ideal, and here’s why:
Sometimes I don’t want to create a .npmrc file and push it to git - in this case I’m out of options.
Now.sh implemented this by also exposing a NPM_RC env variable, where you can put the whole contents of the file. This is nice because I don’t want devs to have .npmrc in their repo when they check out the repo - I just want the build server to have a special npm configuration via the NPM_RC variable.
Any chance we could see this implemented? Thanks a lot!
wire things so that our automatic installation of things doesn’t happen (maybe add an NPM_FLAGS of --dry-run ?)
have your build check for a variable you set only at netlify (or you can just check for $NETLIFY or other variables we set at build time: Build environment variables | Netlify Docs
use that to “gate” your build on first constructing .npmrc correctly based on variables in the account, or use the default settings instead if it is a local build.
But to your point of making the build more configurable, that is the goal of our new build plugins system: Netlify Build Plugins: add powerful capabilities to every build where you can more easily make your dreams a reality and I think that is the best path to get things working to match your precise needs, since we won’t make very big changes to our existing production (soon to be legacy) less configurable build system.
Thanks for the tips. I was trying to create something along the same lines - to generate the correct .npmrc file (only when a custom $NETLIFY variable was set) - but was doing that in the preinstall npm script, which was already too late.
Will try your proposal and come back with feedback - thank you.
Also, thanks for the heads-up with the new build plugins, already requested an invite for that.
When i’m trying npm install --dry-run in order to get packages from a private npm registry, the command will still fail - so dry run is not a solution (when I try --dry-run locally it succeeds when I’m logged into the private registry, but fails when I’m not logged in - so it will obviously fail in the netlify build process as well).
Darn, we thought this would be the solution but it indeed doesn’t seem to work. My guess is the npm process won’t pick up any more changes to the configuration once it has started running.
Our team figured out a *cough* slightly hacky *cough* way to get this to work with the npm preinstall script.
Create a preinstall script (for example at ./scripts/netlify-preinstall.js):
const fs = require('fs')
const { spawnSync } = require('child_process')
// Netlify does not support Github Packages (or other private package registries besides npm), options are:
// - Commit .npmrc to repo - However, now we have a secret token inside our repo
// - Environment variable in .npmrc - However, this requires all developer machines to have the same environment variable configured
// - Get creative with the preinstall script... :)
// Only run this script on Netlify
if (process.env.NETLIFY === 'true') { // this is a default Netlify environment variable
// Check if .npmrc already exists, if it does then do nothing (otherwise we create an infinite yarn loop)
if (!fs.existsSync('.npmrc')) {
// Create .npmrc
fs.writeFileSync('.npmrc', `//npm.pkg.github.com/:_authToken=${process.env.GITHUB_TOKEN}\n@oliverit:registry=https://npm.pkg.github.com/\n`)
fs.chmodSync('.npmrc', 0o600)
// Run yarn again, because the yarn process which is executing
// this script won't pick up the .npmrc file we just created.
// The original yarn process will continue after this second yarn process finishes,
// and when it does it will report "success Already up-to-date."
spawnSync('yarn', { stdio: 'inherit' })
}
}
Configure npm/yarn to run this script before installing dependencies by defining the preinstall script inside the package.json file: