Memory Usage & Puppeteer - function, or repo-based?

Hi all,

first time posting (so apologies if I get something wrong). I’m trying to get https://bitsofco.de/how-to-use-puppeteer-in-a-netlify-aws-lambda-function/ working, but I think I’m exceeding the lambda-functions memory limit. I’m elaborating on this Puppeteer demo by trying to get it working within a basic Gatsby site, but my “screenshot” function states “Memory Usage: 85 MB” when I try and hit the endpoint, and it fails with “ENOENT: no such file or directory, open ‘//…/bin/chromium.br’”.

However, a separate, basic “Hello, world!” function within the same repo (which works) is saying “Memory Usage: 77 MB” (which seems massive for what it’s doing).

Even this would exceed the supposed 50MB memory allocation, right? So why does it still work?

Is the memory usage of each function based on that individual function, or whatever else is being installed as part of the package.json?

I don’t think there is a 50Mbyte allocation limit; we pin you at 1Gb mem usage in lambdas.

The limitations folks usually run into are usually on function size on disk (50/250 zipped/not), not memory used.

That error message also doesn’t sound like an “out of memory” but rather a missing file.

If you have a successful deploy and URL for the function and could share the link to the logs and the URL for the function for us to look at, that’d be a good start for us to try to see what is happening :slight_smile:

Also would be useful to know if you can run locally using netlify-lambda serve or netlify dev ?

Hi @fool. Thanks a lot for replying.

Repo is here: https://github.com/coderkind/gatsby-lambda-screenshot-demo
Function is here: https://xenodochial-newton-c1604d.netlify.com/.netlify/functions/screenshot

I haven’t gotten the lambda to work locally yet.

The function log for the above says:
7:23:23 PM: 2019-12-03T19:23:23.721Z 102894ba-3022-438c-98c7-0ef066d4fd5f {“errorMessage”:“Failed to load ‘iltorb’ package.”,“errorType”:“Error”,“stackTrace”:[“Function.get brotli [as brotli] (/var/task/screenshot.js:8:48749)”,“Promise (/var/task/screenshot.js:8:49698)”,“new Promise ()”,“inflate (/var/task/screenshot.js:8:49327)”,“Function.get executablePath [as executablePath] (/var/task/screenshot.js:8:47812)”,“t.handler (/var/task/screenshot.js:1:345908)”]}
7:23:23 PM: Duration: 9.39 ms Memory Usage: 69 MB Init Duration: 126.32 ms
7:28:00 PM: 2019-12-03T19:28:00.030Z f264db1c-f84a-4229-8f08-a4ea2b649bbb {“errorMessage”:“spawn ETXTBSY”,“errorType”:“Error”,“stackTrace”:["_errnoException (util.js:1022:11)",“ChildProcess.spawn (internal/child_process.js:323:11)”,“Object.exports.spawn (child_process.js:502:9)”,“e.exports.launch (/var/task/screenshot.js:8:12789)”,""]}
7:28:00 PM: Duration: 32.61 ms Memory Usage: 69 MB

error seems pretty straightforward:

“Failed to load ‘iltorb’ package.”

You are clearly not including it in your function, nor in your package.json in that repo.

so…try including it?

Unfortunately not (but just added it again); I’ve played around with including this in the past. I’ve added and deployed things with it in once more, but a different error now from https://xenodochial-newton-c1604d.netlify.com/.netlify/functions/screenshot - “spawn ETXTBSY”

@silentfail can you try deploying https://github.com/netlify-labs/netlify-functions-headless-chrome and see if that works for you

@futuregerald no luck. Function log:

1:09:22 AM: 2019-12-10T01:09:22.111Z 0e3372d5-7033-4f3b-8f4b-7cb68e2bcc41 INFO spawning chrome headless
1:09:24 AM: 2019-12-10T01:09:24.614Z 0e3372d5-7033-4f3b-8f4b-7cb68e2bcc41 INFO error Error: Failed to launch chrome!
/tmp/chromium: error while loading shared libraries: libnss3.so: cannot open shared object file: No such file or directory

TROUBLESHOOTING: https://github.com/GoogleChrome/puppeteer/blob/master/docs/troubleshooting.md

at onClose (/var/task/node_modules/puppeteer-core/lib/Launcher.js:348:14)
at Interface.<anonymous> (/var/task/node_modules/puppeteer-core/lib/Launcher.js:337:50)
at Interface.emit (events.js:215:7)
at Interface.close (readline.js:403:8)
at Socket.onend (readline.js:180:10)
at Socket.emit (events.js:215:7)
at endReadableNT (_stream_readable.js:1183:12)
at processTicksAndRejections (internal/process/task_queues.js:80:21)

1:09:24 AM: Duration: 2517.40 ms Memory Usage: 398 MB Init Duration: 278.64 ms
1:09:41 AM: 2019-12-10T01:09:41.314Z f3436fb2-1430-471a-a53c-a25d75723c9b INFO spawning chrome headless
1:09:41 AM: 2019-12-10T01:09:41.349Z f3436fb2-1430-471a-a53c-a25d75723c9b INFO error Error: Failed to launch chrome!
/tmp/chromium: error while loading shared libraries: libnss3.so: cannot open shared object file: No such file or directory

TROUBLESHOOTING: https://github.com/GoogleChrome/puppeteer/blob/master/docs/troubleshooting.md

at onClose (/var/task/node_modules/puppeteer-core/lib/Launcher.js:348:14)
at Interface.<anonymous> (/var/task/node_modules/puppeteer-core/lib/Launcher.js:337:50)
at Interface.emit (events.js:215:7)
at Interface.close (readline.js:403:8)
at Socket.onend (readline.js:180:10)
at Socket.emit (events.js:215:7)
at endReadableNT (_stream_readable.js:1183:12)
at processTicksAndRejections (internal/process/task_queues.js:80:21)

1:09:41 AM: Duration: 37.48 ms Memory Usage: 398 MB
1:09:50 AM: 2019-12-10T01:09:50.096Z b668f5af-9f5c-4508-8517-752a2ecee409 INFO spawning chrome headless
1:09:50 AM: 2019-12-10T01:09:50.108Z b668f5af-9f5c-4508-8517-752a2ecee409 INFO error Error: Failed to launch chrome!
/tmp/chromium: error while loading shared libraries: libnss3.so: cannot open shared object file: No such file or directory

TROUBLESHOOTING: https://github.com/GoogleChrome/puppeteer/blob/master/docs/troubleshooting.md

at onClose (/var/task/node_modules/puppeteer-core/lib/Launcher.js:348:14)
at Interface.<anonymous> (/var/task/node_modules/puppeteer-core/lib/Launcher.js:337:50)
at Interface.emit (events.js:215:7)
at Interface.close (readline.js:403:8)
at Socket.onend (readline.js:180:10)
at Socket.emit (events.js:215:7)
at endReadableNT (_stream_readable.js:1183:12)
at processTicksAndRejections (internal/process/task_queues.js:80:21)

1:09:50 AM: Duration: 18.38 ms Memory Usage: 398 MB

Can’t really say what’s going on there specifically:

/tmp/chromium: error while loading shared libraries: libnss3.so: cannot open shared object file: No such file or directory

…other than that is a shared library that did not seem to get packaged up with your function. You’re saying that the one that Gerald asked you to deploy doesn’t work for you? What happened when you tried? I don’t think the same error would have shown up there since that setup should work for everyone without the error you show.

Also in general - you’ll need to get a function working locally before it can work here, so that is always step 0 before trying to deploy.

I’ve since managed to get things working locally, but still getting errors on Netlify (ENOENT: no such file or directory, open ‘//…/bin/chromium.br’).

Gerald’s example didn’t work for me, no.

I can’t see anything screamingly obvious from the function log as to what’s going wrong.

Can you see how much memory your function is consuming when you get it running locally? I’m curious if it needs 1.5gb since we provide 1gb in functions.

I’ll check the docs, but is there a way netlify-dev can tell you?

You could probably do something like run top on your unix-based development machine to watch how much memory the process is using in realtime. Or, some of the options in this thread may work: https://stackoverflow.com/questions/774556/peak-memory-usage-of-a-linux-unix-process . I don’t know an equivalent on windows. I don’t think netlify dev will provide you much detail on memory usage natively.