Functions and node_modules

I’m testing locally with netlify dev and my function works perfectly. When I commit to Github, my deploy fails because it can’t find a required dependency under functions/myfunc. Obviously I’ve got node_modules in .gitignore and I assumed that Netlify would run npm i automatically on build for functions. Since it isn’t, how do I correct this?

Hi @cfjedimaster,

One option is to include your dependencies in your root package.json. If you have a second package.json in your functions directory then you can chain their installation as part of your build command using &&.

1 Like

Ok… so… are you saying use a package.json at project root with a script that runs npm i for the function subdirectory?

Also - I tried adding netlify-lambda to my project. In my root package.json, I then tried netlify-lambda build functions'. This is documented here: https://www.netlify.com/docs/functions/#tools-for-building-javascript-functions. Doing so gives me a new error:

netlify-lambda Error: Function source folder (specified in netlify-lambda serve/build command)
9:18:39 AM:       and publish folder (specified in netlify.toml)
9:18:39 AM:       should be different.

As an update - I’ve now tried specifying ./functions in my site settings. I have that same value in netlify.toml, but maybe it will help? Currently my deploy is enqueued so I’m waiting.

hi raymond!

Ok… so… are you saying use a package.json at project root with a script that runs npm i for the function subdirectory?

yes. infact i like to do this as a prebuild or preinstall hook:

{
  "scripts": {
    "prebuild" : "cd functions/myfunction && npm i",
   "build": "gatsby build"
  }
}

you can also stick it inside the netlify.toml command field, but thats really up to personal preference. bottom line is, you’re responsible for writing the bash commands to install any dependencies that aren’t at the root.

i’m well aware this isn’t ideal. this may change in future iterations of the build bot that we are currently prototyping. but its what we have for now.

Also - I tried adding netlify-lambda to my project. In my root package.json, I then tried netlify-lambda build functions '. This is documented here: https://www.netlify.com/docs/functions/#tools-for-building-javascript-functions. Doing so gives me a new error:

this is almost a different topic altogether. to use netlify-lambda, be sure to read the docs: “At a high level, netlify-lambda takes a source folder (e.g. src/lambda , specified in your command) and outputs it to a built folder, (e.g. built-lambda , specified in your netlify.toml file).”

given this it would evidently be a bad idea to have your source and dist folders be the same folder. this is what the error is warning you about. you’re coming from a netlify dev workflow, which has no build step, so you don’t have a source and dist folder. For an easy fix, change your functions key inside netlify.toml to something else.

netlify-lambda serves slightly different and overlapping needs than netlify dev. I did my best to explain in the docs: https://github.com/netlify/netlify-lambda#netlify-lambda “Netlify Dev vs Netlify-Lambda”

1 Like

This will be updated with a future version of our build system.

In the meantime you can use this script that will automatically loop through your functions directories and ensure that the dependancies are installed.

and run this in the prebuild scrip of your package.json like so:

  "scripts": {
    "prebuild": "node ./scripts/installer.js",
    "build": "react-scripts build"
  },
1 Like

Thank you both. I was in a meeting the last 40 minutes or so and was able to quickly test.

So first - yeah, netlify-lamda was a mistake on my part. I was just throwing crap at the wall to see what would stick. :wink: I’m going to read your link, but it seems that for how I use Netlify Functions, I can probably avoid it.

Running npm i on the subdirectory worked great, and David, your script to do this dynamically is cool too.

I’m going to write this up in a blog post to specifically cover the use case of:

  • using netlify dev locally to test
  • using the CLI to scaffold an app
  • using code that requires node modules (which will be most functions)
  • and using Git to deploy where you are not committing node_modules

It sounds like yall may have a way to make this easier in the future too, which is good. Honestly this feels like something I’d have trouble documenting, but until you do get the fix working, maybe a quick reminder at the end of https://docs.netlify.com/functions/build-with-javascript/? Before you get into “Tools for Building JavaScript Functions”. And actually, isn’t that section a bit out of date now with the way the netlify function can scaffold an application and netlify dev can run it for you?

Finally - thank you all!

1 Like

Hi @DavidWells, is this installer.js necessary, or is it now included in the build system?

It’s not available yet, and won’t be for you for at least a little while. We’ll post more widely about the new build system as things develop :slight_smile:

I just wanted to share here that the new and improved Netlify docs include a couple improvements relevant to this discussion.

So is it fair to say, given serverless functions A, B, C:

A needs foo, goo npm dependencies
B needs zoo
C needs foo, roo

If I use a root package.json using foo, goo, zoo, roo, it will then handle installing what I need?

But wait - that would install to my site root node_modules, not relative to my functions.

How would you handle this?

1 Like

Hi @cfjedimaster, as long as you require the dependencies in your lambda functions, what you describe should be handled correctly. Specifically, a file structure like:

/functions/foo.js
/package.json
/node_modules/

Our buildbot uses zip-it-and-ship-it to handle your dependencies, which is flexible enough to handle dependencies in either your root package.json and/or a package.json in your functions folder.

Let me know if that answers our question. Thanks!

Well, this part: “and/or a package.json in your functions folder” to me means you made it work as easy as possible. So given functions/foo/node_modules and functions/goo/node_modules, both functions requiring what they need and such, Netlify will run npm i for both folders correctly?

The buildbot won’t actually do the npm i except for the root package.json, based on the code as far as I can tell. You will need to do the install in each of those function subfolders individually (as part of your build command). I think maybe it makes more sense to have one package.json in the function folder that will have the dependencies collectively so you only need to run the npm i once.

So… how does that work then? If I need node-fetch in /functions/goo, do I installed it in /functions/ and in goo’s code require(’…/node-fetch’) or some such? Sorry if I’m missing the obvious here.

1 Like

Yup, you’ll make sure the node-fetch package is in the package.json inside your functions/ folder. During your build command, you’ll need to make sure to run npm i in that folder as well (e.g. cd functions/ && npm i && <rest of your build command>).

Alternatively, you can have node-fetch in your main package.json file so you don’t need to add those commands in your build command.

Lastly, you’re correct that you’ll need to have the require statement as well.

Hopefully that filled in what you were missing. Let me know if it doesn’t.

Hmm. So probably the most ideal solution is, at site root, npm i my needs, and in my functions/X folders, use require(’…/…/SOMETHING’), right?

Yea, that sounds about right. You might not even need to use the absolute path as our buildbot should traverse your repo for package.json files to resolve that module.

I can’t get node modules to work in the functions. Help would be realy appreciated.

These are the steps I have taken so far:

Folder structure

/functions/submission-created.js
/package.json
/node_modules/

I do not have the installer.js nor package.json in functions folder. Also I do not run the prescript.

My goal is to make it work like @cfjedimaster using require(’…/…/SOMETHING’)

I want to use the googleapis. googleapis is in dependencies of package.json in root folder

 "dependencies": {
    "googleapis": "^45.0.0",    
  },

I am failing to require the googleapis from root out of my functions/submission-created.js. The function gets called but fails. I get one of those following errors depending on what I am trying.

 Unable to import module 'submission-created': Error
    at Function.Module._resolveFilename (module.js:547:15)
    at Function.Module._load (module.js:474:25)
    at Module.require (module.js:596:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/var/task/submission-created.js:1:78)
    at Module._compile (module.js:652:30)
    at Object.Module._extensions..js (module.js:663:10)
    at Module.load (module.js:565:32)
    at tryModuleLoad (module.js:505:12)
    at Function.Module._load (module.js:497:3)

the other error:

{"errorMessage":"Cannot find module 'googleapis'","errorType":"Error","stackTrace":["Function.Module._resolveFilename (module.js:547:15)","Function.Module._load (module.js:474:25)","Module.require (module.js:596:17)","require (internal/module.js:11:18)","exports.handler (/var/task/submission-created.js:6:18)"]}

I tried inside of sumbission-created.js :

var {google} = require('.../googleapis');
var {google} = require('../googleapis');
var {google} = require('/googleapis');
var {google} = require('googleapis');

Looking forward to your answers.

Thank you guys.

@Dergarcon can you share the code that’s in your function? If your repo is public, it would be useful to see it. Thanks.