Node Externals and other flags required to build successfully because of zip-it-and-ship-it's bugs cost time and money

Hello there,

Because of issues with netlify’s zip it and ship it and other build components, I am forced to use nodeExternals() and other configuration settings in order to get my build to pass. This off course rebuilds every function from scratch, bypassing the intelligent rebuild only what’s changed process that netlify normally would. This is of course, not my fault. It unfortunately leads to longer build times, 3 x 4 times longer depending on the number of functions, that did not change in that commit at all, netlify rebuilds and deploys again from scratch, because of required configuration, like nodeExternals() and global.Gently. Is there a solution to this tension between actually publishing functions and paying for unnecessary build minutes?

Sean

Hey Sean, sorry to hear you are not having the experience you’re hoping for. Before we weigh in on the cost issue, can you say a little more about what issues you are experiencing?

Maybe there is something we can do about that first. The more details the better!

Perry,

I’m using typescript with netlify functions.

I cannot get my functions to compile at all without nodeExternals() and global.Gently

Does netlify dev support typescript yet?

Would generating functions as I’ve seen Shawn do with terminal commands be better than having files like this:

image

Would you like me to create a public repo so you can investigate?

hi @seanaguinaga - thank you for your patience. It would actually be helpful for us to to see a public repo - we’re not 100% sure we’ll have a recommendation for you, but seeing how you are putting things together could spark some ideas as we chat more about this internally.

Perry, here’s that public repo.

This is based of of Shawn’s repo.

In the package.json you can see the two different lambda commands for serving/building.

Something as simple as using graphql causes this error without nodeExternals() for example.

There are more modules that fail without globalGently, but i thought I’d limit it to node externals.

With nodeExternals, it seems netlify’s build process will always rebuild every function, regardless of whether a function changed or not, resulting in more build time, and more $$$ for Netlify and customers.

Please try using the run start____ with either config or no config for the lambda functions, commenting out options from the webpack config files to see what I mean here.

TypeScript would ideally be a 1st-class citizen within netlify’s build process.

If this can be avoid by using another method please advise! I really appreciate you looking into this.

I will add more and more examples of netlify’s build process failing without explicit config to this repo as time goes on.

Note that this fails with/without babelrc in the lambda folder or other files like that. it’s unrelated but just improves dev ergos, like adding optional chaining and things like that but has no effect on netlify’s build process failing, but of course, you can test that to verify if you’d like.

sean - many thanks.

We’re definitely interested in understanding what is happening here. Realistically, this may not happen until next week, but I am leaving this question open so we don’t forget about this. We’ll respond here on this thread as soon as we have had a chance to dig into this a little bit.

thank you again for taking the time to write this up and share - it’s contributions like this that allow us to identify where we can do better. :muscle:

Hi @seanaguinaga!

I’m trying to unwind some of the stuff happening with your setup.

  • Your build command is npm run build
  • The npm script for that is run-p build:**
  • This will run these scripts in parallel:
    • build:app
    • build:lambda
    • build:lambda-with-config

From this I get the notion that the process might be building your functions twice at the same time.
This can cause a lot of weird behaviour.
As a first step, I can recommend to make sure your functions are only built once.

Some other notes:

  • When a build happens, your functions will always be built. The only way to prevent that is using the ignore setting to not run the build at all.
  • Only after your functions have been built we will diff them against the existing ones. If you have ideas how to improve that diffing, we’re all ears.
  • It should be pretty usual for nodeExternals to be required for a build of functions running on AWS lambda (which Netlify functions do)