[Common Issue] Using private NPM modules on Netlify

Thanks for the tips. I was trying to create something along the same lines - to generate the correct .npmrc file (only when a custom $NETLIFY variable was set) - but was doing that in the preinstall npm script, which was already too late.

Will try your proposal and come back with feedback - thank you.
Also, thanks for the heads-up with the new build plugins, already requested an invite for that.

1 Like

When i’m trying npm install --dry-run in order to get packages from a private npm registry, the command will still fail - so dry run is not a solution (when I try --dry-run locally it succeeds when I’m logged into the private registry, but fails when I’m not logged in - so it will obviously fail in the netlify build process as well).

1 Like

Darn, we thought this would be the solution but it indeed doesn’t seem to work. My guess is the npm process won’t pick up any more changes to the configuration once it has started running.

Our team figured out a *cough* slightly hacky *cough* way to get this to work with the npm preinstall script.

Create a preinstall script (for example at ./scripts/netlify-preinstall.js):

const fs = require('fs')
const { spawnSync } = require('child_process')

// Netlify does not support Github Packages (or other private package registries besides npm), options are:
//   - Commit .npmrc to repo - However, now we have a secret token inside our repo
//   - Environment variable in .npmrc - However, this requires all developer machines to have the same environment variable configured
//   - Get creative with the preinstall script... :)

// Only run this script on Netlify
if (process.env.NETLIFY === 'true') { // this is a default Netlify environment variable
	// Check if .npmrc already exists, if it does then do nothing (otherwise we create an infinite yarn loop)
	if (!fs.existsSync('.npmrc')) {
		// Create .npmrc
		fs.writeFileSync('.npmrc', `//npm.pkg.github.com/:_authToken=${process.env.GITHUB_TOKEN}\n@oliverit:registry=https://npm.pkg.github.com/\n`)
		fs.chmodSync('.npmrc', 0o600)
		// Run yarn again, because the yarn process which is executing
		// this script won't pick up the .npmrc file we just created.
		// The original yarn process will continue after this second yarn process finishes,
		// and when it does it will report "success Already up-to-date."
		spawnSync('yarn', { stdio: 'inherit' })
	}
}

Configure npm/yarn to run this script before installing dependencies by defining the preinstall script inside the package.json file:

  "scripts": {
    "preinstall": "node ./scripts/netlify-preinstall.js"
  },

Don’t forget to add the GITHUB_TOKEN variable in your Netlify team or site settings.

1 Like

Almost there… I was doing an almost identical approach, without the if check, so always writing the file sync, but without the chmod and spawnsync.

Since I previously ran the script with some errors, I already have .npmrc - the “clear cache and deploy site” button doesn’t seem to delete the existing .npmrc file.

after a few more tries, and console.logging, I see that npmrc is overriden on each build, setting the registry back to the default npm registry //registry.npmjs.org/:_authToken=… my token is correct here, but the file is overridden with the default npm registry.

Also, writing the file without the if check and spawning yarn again just triggers an infinite loop, so I have to cancel the deploy

I think the easiest thing to do, if the netlify team would be kind enough :pray::pray::pray: would be to introduce a special NPM_RC variable with the contents to be persisted in the .npmrc file.

@radu, I’d be happy to create a feature request for this. Before I do so, I want to be 100% sure your expectations for the feature. Also, I want to see if there is one more way to workaround this.

Regarding the solution, if I understand correctly (and if I’m wrong please let me know), the solution would work like this:

  • the NPM_RC file would be defined in the Netlify UI
  • at build and deploy time, Netlify will copy the contents of this environment variable into the .npmrc file
  • this can then be used by npm install during the dependency install phase of the build and deploy

Regarding the workaround, it should be possible to manually do this now. Possibly by echoing the contents of the environment variable to this file in the build commands and then re-running the npm install.

Would you confirm if the feature request description above matches your requirements? Also, if you do test the workaround above, please let us know if it works for you or not.

1 Like

@luke That’s exactly how I think it should work :+1:

Regarding the workaround above, I couldn’t get it to work - here’s what’s happening:

  • I intentionally remove the if check above, just to force a write to .npmrc. I’m console.logging the value and it’s the new, correct value that I need. Yarn goes into infinite loop, as expected and explicitly mentioned in the comments, so I cancel the build
  • next time, I push an update that includes the if check, so we no longer get an infinite loop - but just before the if check, I console log the value in the .npmrc file and it’s no longer the value I wrote last time (at the previous build), but rather it points to the default npm registry//registry.npmjs.org/:_authToken=.... (this is the contents of the file) at this point

I’ve also tried “clear cache and deploy site” button, but same results - the .npmrc file seems to be overridden each time.

Thanks for considering this :pray:

OK, thanks for talking through that with us and trying all the things. I’d post a link to this thread and description of your problem and hoped for solution here as a feature request:

I do think that the advice will probably be “WONTFIX but see the build plugins beta where you can implement this yourself” (in a couple of months), but instead of us playing telephone to our build system developers probably better to let you represent yourself (and see any responses they send to you :))

Let me know if you can’t do that for some reason and I can do it for you and will still link you to the open issue to track follow-ups there.

@radu any chance you still have a .npmrc file checked into your git repo? I am unable to reproduce your issue with our script, but having an existing .npmrc file inside your repo might be the reason you are having this problem.

Another option would be to replace the check if a .npmrc file exists with a new environment variable which is set when spawning yarn from a child process. In that case the script would look something like this:

const fs = require('fs')
const { spawnSync } = require('child_process')

// Netlify does not support Github Packages (or other private package registries besides npm), options are:
//   - Commit .npmrc to repo - However, now we have a secret token inside our repo
//   - Environment variable in .npmrc - However, this requires all developer machines to have the same environment variable configured
//   - Get creative with the preinstall script... :)

// Only run this script on Netlify
if (process.env.NETLIFY === 'true') { // this is a default Netlify environment variable
	// Check if .npmrc was already generated by this script. If it does then do nothing (otherwise we create an infinite yarn loop)
	if (process.env.NETLIFY_NPMRC_DONE !== 'true') {
		// Create .npmrc
		fs.writeFileSync('.npmrc', `//npm.pkg.github.com/:_authToken=${process.env.GITHUB_TOKEN}\n@oliverit:registry=https://npm.pkg.github.com/\n`)
		fs.chmodSync('.npmrc', 0o600)
		// Run yarn again, because the yarn process which is executing
		// this script won't pick up the .npmrc file we just created.
		// The original yarn process will continue after this second yarn process finishes,
		// and when it does it will report "success Already up-to-date."
		spawnSync('yarn', { stdio: 'inherit', env: { ...process.env, NETLIFY_NPMRC_DONE: true } })
	}
}
1 Like

@rvanmil :tada: thanks for the solution it’s working! Was thinking about doing something similar, but didn’t know I can send new env vars to spawn - that was the missing piece of the puzzle.

Regarding your question - no, I don’t have a checked-in npmrc file - that’s what I wanted to avoid.

@fool as this is now fixed, and since I just got access to the beta build plugins ( thanks Netlify team :heart:), I will try and experiment with the new build system, as I’m convinced I can solve the problem much more elegantly with that.

4 Likes

This is solution is a non-starter for us. Committing a local npmrc file breaks a ton of ecosystem workflows and requiring that everyone on the team have their token in an env var is an unreasonable pain point, and not how the ecosystem works generally. If Netlify wants to continue the (nice) experience of auto installing deps then it should automatically used the configured NPM_TOKEN when installing, or allow users to override at run time

1 Like

Sorry to hear you don’t like the available solution. It’s working for a lot of folks, so probably not a high priority for our team to change things right now, but would you mind outlining your perfect solution here so the wider team can consider it?