Folder structure of function deployment differs

Hey,

I might be going mad over here, so better ask here:

I have a lambda function foo that serves a file from the same directory as the function file. Upon deployment the file sometimes seems to live right next to the function Javascript file (whose folder path I can get via process.env.LAMBDA_TASK_ROOT), sometimes in ./src, sometimes in ./src/foo or even ./src/functions/foo.


I of course used Google and this community to search for other examples, but depending on which post you look at you will see any of the above options (or even more!) suggested. (e.g. function-deploy-test/zipped-function.js at master Β· netlify/function-deploy-test Β· GitHub from @Dennis which suggests ./foo as the location)


This is one deployment, where src works:
GitHub - janpio/netlify-functions-file-foo
https://netlify-functions-image-foo.netlify.app/logo.png

INFO	process.env.LAMBDA_TASK_ROOT /var/task

INFO	# content of process.env.LAMBDA_TASK_ROOT
INFO	[ 'image.js', 'src' ]

INFO	# content of process.env.LAMBDA_TASK_ROOT/src
INFO	[ 'image.js', 'logo.png' ]

INFO	# content of process.env.LAMBDA_TASK_ROOT/src/functions
INFO	undefined

INFO	# content of process.env.LAMBDA_TASK_ROOT/src/functions/image
INFO	undefined

And this is the other deployment with pretty much the same code (same folder structure etc), but totally different folder structure on the deployment:

INFO	process.env.LAMBDA_TASK_ROOT /var/task

INFO	# content of process.env.LAMBDA_TASK_ROOT
INFO	[ 'image.js', 'src' ]

INFO	# content of process.env.LAMBDA_TASK_ROOT/src
INFO	[ 'functions', 'node_modules' ]

INFO	# content of process.env.LAMBDA_TASK_ROOT/src/functions
INFO	[ 'image' ]

INFO	# content of process.env.LAMBDA_TASK_ROOT/src/functions/image
INFO	[ 'image.js', 'logo.png' ]

Locally with netlify dev the files are directly in process.env.LAMBDA_TASK_ROOT.


Any idea what is going on here?

I am a bit scared that I am doing something fundamentally wrong or the deployments will change their behavior and break my site :confused:

hey there, i will ask our functions expert to weigh in - stay tuned.

Hi! I’ve not tested this but I think it all depends on how you deploy your function. Differences between single-file function (like what you have), function in a folder, and self-zipped. So it makes sense that image file is in the same folder as your single-file function. If it was a function-in-a-folder style deploy, then the image would be inside the folder, like in my example.

Let me know if that makes sense.

Unfortunately not, as they are both pretty much identical. Both deployments have 1 function in a subfolder of functions - and their deployment structure is different. (Happy to share the actual deployment via private message if you want to look into them and the difference)

Do you have example sites that you can share out in the open that shows the difference where the same function was behaving differently? Note that netlify-lambda will work differently from a deployed function as it’s mainly used to test general function logic rather than emulating every minutiae.

You can compare https://netlify-functions-image-foo.netlify.app/logo.png to https://images2.prisma.io/footer-logo.png. Under the hood they both execute a function functions/image/image.js that tries to server a file from the same folder. To make it work in both cases I needed to use a different path - see the log output in my original message. (The sourcecode of the second one is unfortunately not public on Github, but I hope you can access the code of the function on the deployment.)

The only thing I can think of is that the deploys were built using different versions of our build-image/buildbot. Could you provide links to specific deploys? The deploy should say what version is used at the beginning of the logs (at the top).

Additionally, the way functions are bundled is determined by the zip-it-and-ship-it package here: GitHub - netlify/zip-it-and-ship-it: Intelligently prepare Node.js Lambda functions for deployment which is what we use in the buildbot. If you provide those specific deploy links, we can also confirm if there was a change in the way a function is bundled that occurred between the two deploys.

1 Like

Of course:

But I checked the logs now, identical versions :frowning:

Only difference:

❯ Loading plugins
   - netlify-plugin-debug-cache@1.0.3 from Netlify app
​
...

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ 3. onEnd command from netlify-plugin-debug-cache β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
​
Saving cache file manifest for debugging...
Cache file count: 0
Cache manifest saved to /opt/build/repo/cache-output.json
Please download the build files to inspect cache-output.json.
Instructions => http://bit.ly/netlify-dl-cache
​
(netlify-plugin-debug-cache onEnd completed in 214ms)

But I added that while already debugging what was wrong.

Ah, and one has Starting to deploy site from '' and the other one Starting to deploy site from '/'. Not really sure where that difference comes from. They are configured the same way in the accounts.

The publish directory difference shouldn’t affect how a function is bundled and deployed. Since I can’t see the repository for β€˜images2-prisma-io’, it’s hard to say why there is a difference. Like you said, those examples have different β€˜deployment structures’.

One thing to note is that my zipped-function example is based on a function that was bundled manually (not by the buildbot’s zip-it-and-ship-it feature). That’s probably the best option to take that would give you a consistent path for your bundled files/images.

What would you need to see from images2-prisma-io? If you send me your GH username via DM I can invite you - I can just not make it public.

hi janpio,

sorry to hear that you are still having difficulties with this.

Unfortunately, with only seven support engineers who provide support for hundreds of thousands of people worldwide, we can’t, unfortunately, go more in depth here than we already have - getting involved in individual git repo’s and exchanging information via DM is outside of the scope of support we can readily priovide.

If you can create a publicly shareable git repo with some dummy code, if the issue is very sensitive, you can post it here and maybe one someone will have more ideas for you!

I investigated further:

As soon as I have the line const { PrismaClient } = require('@prisma/client') in my function, the path of the images change. Will create a super simple reproduction with that.

Ok, the explanation was even simpler:
If you embed a library that needs to be installed into node_modules in the function, the file structure of the system is different.


Here is the reproduction:

The repository contains a no-node_modules.js and a with-node_modules.js function.
Here is some logging output that shows their file system structure:

no-node_modules.js:

INFO	process.env.LAMBDA_TASK_ROOT /var/task
INFO	# content of process.env.LAMBDA_TASK_ROOT
INFO	[ 'no-node_modules.js', 'src' ]
INFO	# content of process.env.LAMBDA_TASK_ROOT/src
INFO	[ 'functions', 'node_modules' ]
INFO	# content of process.env.LAMBDA_TASK_ROOT/src/functions
INFO	[ 'no-node_modules' ]
INFO	# content of process.env.LAMBDA_TASK_ROOT/src/functions/no-node_modules
INFO	[ 'netlify.png', 'no-node_modules.js' ]

with-node_modules.js:

INFO	process.env.LAMBDA_TASK_ROOT /var/task
INFO	# content of process.env.LAMBDA_TASK_ROOT
INFO	[ 'src', 'with-node_modules.js' ]
INFO	# content of process.env.LAMBDA_TASK_ROOT/src
INFO	[ 'netlify.png', 'with-node_modules.js' ]
INFO	# content of process.env.LAMBDA_TASK_ROOT/src/functions
INFO	undefined
INFO	# content of process.env.LAMBDA_TASK_ROOT/src/functions/with-node_modules
INFO	undefined

Different although the function code is almost identical but that one line.


So he answer to my initial question is:
As soon as you use a library that needs a node_modules folder in the function deployment, you will have a different file system than is documented and mentioned in most Netlify Community posts.

I am not sure this is intended on Netlify side. If it is, it should probably be documented in some way.

Thanks for the update! I’ve asked our team to look at this and see if it can possibly added to our documentation.

1 Like