Environment Vars Feature requests / Filter env vars to include in Lambda environment parameter

[rob]

Thanks for the replies everyone. Can anyone speak to whether or not there will be action on this in the near future? (within 12 months)

[perry]

hey rob, i wish i could give you a better answer, but i’m not sure, unfortunately

[rob]

@perry @hrishikesh @fool

We’ve recently hit the limit of our work around implemented back when this feature request was opened and now face a more drastic solution because there has not been any progress on this request. I will outline details below and welcome any feedback as well as an update from Netlify engineers on progress.

To summarize where we are at the moment:

• Our combined set of env vars needed for the combined set of Netlify Functions now exceeds the maximum 4k imposed by AWS for all Lambdas.
• We have trimmed env var names and values, migrated non-secret env vars to in-app config files where we can to keep it under 4k but that methodology has reached it’s limit. What is left is secrets only (that we do not want in the repo) that now exceed the 4k limit in total.
• No single Netlify Function needs all 4k of these evn vars. If we could provide a whitelist of env vars for each Netlify Function, our problem would go away.
• I’ve experimented with build plugins that remove large env vars from the process.env object in hopes the modified process.env object was being passed to the deploy step but alas it is not. Removal of larger env vars after the build step completed did not prevent those env vars from being included in the AWS Lambda deployment/upload step. (Specifically what I tried was a modification to the GitHub - bencao/netlify-plugin-inline-functions-env: Inline build time environment variable values into netlify function code so that it becomes available at runtime plugin to first inline the large env vars then remove them from the process.env object after inlining was completed)

Next Steps
Our next steps are to begin a process of re-architecting our sites. We are going to move some functions out of the site into a new Netlify site with only a functions folder (no front-end). This will come with some CORS headers that will need to be added I believe.

I would like to avoid this heavy handed work around because the added complexity it introduces to the developer experience is high. Here are some challenges this introduces for our developers:

• Having to know that some Netlify functions for a site are in in-fact in a different site altogether.
• The code for each of the functions spread across different github repos.
• The deployments have a chance to be out of sync with each other if one is deployed late or one of the two deployments fails.
• Netlify Dev command line will not work for any of the functions moved to the secondary site.

As you can see from the list above, the issues created by the workaround of moving some Netlify functions to their own site are lengthy and seriously degrade the value proposition of Netlify and the Netlify CLI tools.

Netlify and the tooling around the platform have been a great accelerator for our development team. That said, I wonder now if we have outgrown it since we have to resort to such out of the box solutions to this.

Since the feature request was initially made back in Jan 2020 (3 years ago), there has not been any effort to tackle it. The revamped environment variables tooling in the Netlify UI add complexity (11 clicks to adjust an env var value now in the UI) but still don’t address this root problem.

I’m left wondering if it’s time to move on to our own pipeline scripts for deployment Lambdas. I’d much prefer to stick to Netlify though as it’s all just so much easier.

Please let me know if we can expect any solution in the near term (within 3-6 months) so that I can make an informed decision about our future.

[fool]

Hey Rob,

I think there has been progress on this feature request since we last talked! Have you seen this post?

Manage scopes and deploy contexts for environment variables | Netlify Blog

As far as I know, this should solve your use case, but please let me know if not!

[rob]

Thanks for the quick reply Chris. I referenced that new feature in my post actually. It does not tackle the problem. I’m open to being corrected if you can help me understand what I’m missing.

[fool]

I don’t see that you did reference that feature in your post, Rob? I see you referenced some 3rd-party build plugins around environment variables, which is not what I am talking about - I am talking about surfacing some of your vars just to functions via the scoping feature in the blog post I mention.

Anyhow, I’ll ask the product manager for the new environment variables scoping experience to sound off here when she gets a chance (she’s on vacation today at least), since as far as I know, that work was intended to solve exactly your use case, and she will at the very least be interested in where the work fell short, if she can’t see a way to fit your use case into the existing system (but I think she’ll see a way :))

[rob]

The revamped environment variables tooling in the Netlify UI add complexity (11 clicks to adjust an env var value now in the UI) but still don’t address this root problem.

What I meant by that was that it adds complexity to defining the env vars but unfortunately the ability to scope a var to a specific Netlify Function doesn’t appear to be included in the new functionality.

The scopes allowed in the new env vars tooling allow for scoping to “all” functions. All is implied unless I’m missing something in the UI that allows me to select which function to specifically scope to.

[luke]

Hi, @rob. About this below:

@rob

We’ve recently hit the limit of our work around implemented back when this feature request was opened and now face a more drastic solution because there has not been any progress on this request. I will outline details below and welcome any feedback as well as an update from Netlify engineers on progress.

Our engineers thought they had built the feature request because this specific requirement below was not made clear to them:

• The scoping of environment variables to functions must be fined-grained to allow for scoping to some individual functions but not all functions for a site.

This was the original request:

@rob

Feature request
Allow a Netlify function to have an override configuration that specifies which env vars to include in the lambda environment parameter when Netlify creates the Lambda on AWS. If there are >4096 env vars for the entire site but only 1KB needed for a function, the function configuration can select just those env vars that make up the 1KB and thus the Lambda deploy would not fail and thus the whole site deploy would not fail.

Your request above definitely does say “a Netlify function”. You were clear. Despite this, in the discussion with the many people on this topic, that detail got missed. Your specific case (the union of all function environment variables exceeding the limit) was not what anyone else was reporting. When the work on the feature request was done, it did not take the “per function scoping” (instead of “all function scoping”) requirement into account.

For most people reporting the issue, it was that unwanted build-only environment variables were exceeding the function limit. For most other people (maybe all other people) being able to filter out the build-only environment variables solved the issue.

I have entered a new feature request which does make the “per function scoping” requirement (not all functions) absolutely clear. Please note, though, that you are currently the only remaining person or organization that we know of encountering the function limit with the new scoping feature (again, which admittedly does lack the per function scoping). So, new feature request or not, I would like to look for other workarounds for this.

The workaround (and, yes, it is not a solution) we suggested above is tried and true. While only a workaround, it is reliable and it does keep the environment variables out of your repos.

In a general example, the workaround is a find & replace of predefined strings in the functions. For example, you could make API key placeholders like so in the function code:

const API_KEY_FOR_SERVICE_A = "API_KEY_FOR_SERVICE_A_PLACEHOLDER"
const API_KEY_FOR_SERVICE_B = "API_KEY_FOR_SERVICE_B_PLACEHOLDER"

The build command would then be modified to find and replace the strings in function files with the values of the environment variables scoped only to the build. There will be no environment variables passed to the lambda at all as all the values are embedded in the code itself (but never in the repo’s version of the code - it only has placeholders).

Using sed for example, the build command below:

npm run build

would have a sed find/replace prefixed to it like so:

sed -i "s/API_KEY_FOR_SERVICE_A_PLACEHOLDER/${API_KEY_FOR_SERVICE_A}/g;s/API_KEY_FOR_SERVICE_B_PLACEHOLDER/${API_KEY_FOR_SERVICE_B}/g" path/to/function.js && npm run build

That is clearly going to get hard to read/manage for large numbers of environment variables using sed on the command line. The use of sed is only an example. You can likely do this much more flexibly and simply in a programming language of your own choosing.

So, it is only a workaround but it is a workaround that will always work. If there are any questions or anything relating to this to discuss further, please feel free to reply here anytime.

[rob]

Luke,

Thanks very much for the thoughtful reply. I can understand how the specifics of this issue were lost in the discussion since other posts here are confusing it with different problems I’m afraid.

As you said, it’s rather specific and I agree it’s not going to be an issue for the vast majority of users. It’s only an issue with us because we have such a large number of Netlify functions with so many secret keys needed across all of them. Frankly it’s only a couple of large certs over 1kB each that spoil the party. Firebase Admin being one of them.

Well I’m looking forward to trying out the new feature to scope env vars to a named function.

Will it be setup as whitelisting or blacklisting or will we be able to do both?
Is possible to scan the code of the function (like is done for node modules) and only send env vars that are referenced? That might be more intuitive.
If it’s a lot more work to add it to the Netlify UI, I’d be fine with a preview release that adds it to the netlify.toml only first.

If it’s a whitelist/blacklist design, I’d suggest that the product manager take a look at how the inline vars plugin works with an include and exclude array, it might be some inspiration. If she would like help brainstorming the solution or testing it just tell her to let me know. I’m happy to use our specific use case as a test bed.

That ‘sed’ solution will work, thank you for it. You’ve given me another idea for a little cleaner workaround though. Basically I could remove the function scope but leave build scope, then use the inline env vars plugin. When I tried to use that plugin yesterday, it inlined the env vars fine but the problem was that the env var was still sent to AWS in Lambda creation putting it over the 4kB even though there were no longer any references to it.

I’ll test it this morning and let you know how it goes.

Thanks,
Rob

[hrishikesh]

I just published a build plugin which can kinda solve this problem:

GitHub - Hrishikesh-K/netlify-plugin-bundle-env: A Netlify Build Plugin to...

Feel free to try it out and let me know.

[rob]

I just finished testing the workaround @luke inspired and it worked well. Here’s how it works for anyone else waiting for the named function env var scoping feature.
.
.
.
.
.

Setting Up the Test

To be clear we are all talking about the same thing, let’s first create the conditions of having environment vars that exceed the 4kB limit causing the site deployment to fail.

Generate a string that is 4096 in length. This will simulate a large cert or something that is needed in one of your Netlify Functions.

Next add that string to an env var on your site through the new env vars ui. We will call it test_long_var for this test.

Add another env var to make sure we are over the limit. For instance a passphrase used in another Netlify Function. Make sure all scopes are enabled for each.

Now try to build and deploy the site and you’ll see that the site deployment fails because of the 4 kB Lamda limit.

ntl deploy --prod --build --skip-functions-cache --debug

Deploying to main site URL...
✔ Ignoring functions cache (use without --skip-functions-cache to change)
✔ Finished hashing 16 files and 2 functions
✔ CDN requesting 0 files and 2 functions
⠸ (1/2) Uploading test... ›   Warning: JSONHTTPError:  422
 ›   Warning: 
{
  "name": "JSONHTTPError",
  "status": 422,
  "json": {
    "errors": "Your environment variables exceed the 4KB limit imposed by AWS Lambda. Please consider reducing them."
  }
}

 ›   JSONHTTPError: Unprocessable Entity

.
.
.
.

The Work Around

Now let’s show the work around for solving this using a combination of env var inlining and the new scoping features to scope the env var out of the functions deployment.

Install the inline env vars build plugin.

Now that the plugin is installed in our site, we can configure it in our netlify.toml file. Add the following to the netlify.toml and specify which env vars to inline into the functions by adding them by name to the include array.

[[plugins]]
package = "netlify-plugin-inline-functions-env"
  [plugins.inputs]
  include = ["test_long_var"]

This plugin will find any references to that env variable (specified as process.env.test_long_var) and replace that in the source code with the actual value at build time.

Here’s our test Netlify Function before…

And here’s our test Netlify Function after…

.
.
.
.
.

Unfortunately that is not enough to fix the problem because even though the reference to the env var was removed, the Netlify deployment process that sends the function to AWS to create the Lambda still passes the env var in that Lambda creation call.

So to fix that, we must remove the env var from the functions scope so that it will be excluded from deployment of the Lambda to AWS.

Fortunately for us, removing the functions scope does not remove the env var from the preBuild stage of the build, so our inline solution above still works.

So with that all in place, let’s run the build and deploy again.

ntl deploy --prod --build --skip-functions-cache --debug

This time we should see the site deploy successfully with our large env var test_long_var successfully inlined into the function’s source code autoatically.

Deploying to main site URL...
✔ Ignoring functions cache (use without --skip-functions-cache to change)
✔ Finished hashing 16 files and 2 functions
✔ CDN requesting 0 files and 2 functions
✔ Finished uploading 2 assets
✔ Deploy is live!

.
.
.
.
.

Caveats & Downsides

Unfortunately when deploying from your local machine the inline env vars plugin modifies the source files in the preBuild stage. This presents as uncommitted changes to your source code. If deploying from your machine, if you have not yet committed and are deploying WIP code, then this could be messy to cleanup mixing your own changes with changes caused by babel during the inlining. I recommend always committing prior to deploying from the command line (really who doesn’t do this?) then just discarding all of the source code changes after.

[rob]

@hrishikesh

GitHub - Hrishikesh-K/netlify-plugin-bundle-env: A Netlify Build Plugin to inject environment variables in Netlify Functions during Netlify Builds.

Yes, I think we just effectively took the same approach but I just used the existing env inlining plugin combined with removal of the functions scope from the env var.

Great minds think alike

[SamO]

Hiya @rob , thanks for your feedback. We appreciate it greatly! (;

[rob]

My pleasure @SamO. I am trying hard not to outgrow Netlify as we scale. It’s really an excellent platform and we appreciate all of y’alls hard work on it.

[hrishikesh]

@rob

Unfortunately when deploying from your local machine the inline env vars plugin modifies the source files in the preBuild stage.

For newcomers to this thread, the new plugin: GitHub - Hrishikesh-K/netlify-plugin-bundle-env: A Netlify Build Plugin to inject environment variables in Netlify Functions during Netlify Builds., handles restoring the modified files back as original as well (plus other benefits of being able to use TypeScript, indexed signatures and destructuring syntax).

[rob]

Oh I did not realize you had those benefits. Excellent, thanks @hrishikesh !

[rob]

@hrishikesh Can you publish that plugin on Discover | Netlify Integrations

[hrishikesh]

I’ve submitted a PR here:

Add Netlify Plugin Bundle ENV v0.1.0

[rob]

Excellent. I’ll watch out for it going public then give it a test.

Rob

[hrishikesh]

I’ve closed the PR as based on an internal discussion, this is no longer the right way to submit new plugins. I’ll work with the team to get this listed there, but this might not happen till next week. Till then, usage with npm + netlify.toml works well.