Are envs set in build plugins available during function runtime?

dthyresson · July 14, 2020, 2:56pm

I wrote a Netlify Build Plugin over the weekend to automatically fetch a config value from Heroku (in this case a DATABASE_URL connection string) to be used in conjunction with Prisma and RedwoodJS

I followed the pattern used by the existing netlify-plugin-contextual-env here:

github.com

cball/netlify-plugin-contextual-env/blob/master/src/index.js#L18


      
          * @param {*} mode the mode to use (prefix or suffix)
          */
          function setEnvWithValue(key, contextOrBranch, mode) {
           const envVar = mode === 'prefix' ? `${contextOrBranch}_${key}` : `${key}_${contextOrBranch}`;
          
          
 if (!process.env[envVar]) {
             return;
           }
          
          
 console.log(`Exporting ${key}=${process.env[envVar]}.`);
           process.env[key] = process.env[envVar];
          
          
 return `${key}=${process.env[envVar]}\n`;
          }
          
          
module.exports = {
           onPreBuild: async ({ inputs }) => {
             const context = `${process.env.CONTEXT}`.toUpperCase().replace(/-/g, '_');
             const branch = `${process.env.BRANCH}`.toUpperCase().replace(/-/g, '_');
             const { mode } = inputs;

and did essentially the same:

github.com

dthyresson/netlify-plugin-heroku-database-url/blob/master/src/index.js#L63


      
              try {
                const options = {
                  appName: inputs.app_name,
                  config: inputs.heroku_config || 'DATABASE_URL',
                }
          
          
      log(
                  `Setting Netlify ENV ${inputs.netlify_env} to Heroku config database connection.`
                )
          
          
      process.env[inputs.netlify_env] = await getDatabaseUrl({
                  options,
                })
              } catch (error) {
                const message = `Unable to update ${inputs.netlify_env} from Heroku config`
                log(message, ERR)
                log(error, ERR)
                utils.build.failBuild(message)
              }
            },
          }

But – while the env is available during the build runtime (and I’ve confirmed this) it is not available to functions that rely on the DATABASE_URL it sets.

I’ve tried with/without setting a value, seeing if the plugin will override, etc.

Note: my function does in fact work when normally setting ENV via Netlify dashboard UI as it finds the env.

And it sort of makes sense - it’s setting it during the “build process”.

But - is there any way to set and save a env setting during the build runtime that is available to functions?

This concept isn’t limited to database urls – I might refresh an API token used by a 3rd party api in a function and want to set it automatically. Lots and lots of use cases.

My only thought is to use the NetlifyAPI and the updateSite to set envs, but I really really really do not want to do that for many many many reasons re: having peoples’ access tokens and the responsibilities there given the power of the NetlifyAPI once you obtain one. See my post on RedwoodJS about that:

Any ideas how to set the env for functions? Did I miss something?

ehmicky · July 15, 2020, 2:59pm

Hi @dthyresson,

At the moment, the only environment variables available in Netlify Functions at runtime are the ones specified in the Netlify app UI. Environment variables set in netlify.toml or set inside a Build plugin with process.env.name = value are not available there unfortunately.

One (arguably hacky) way to workaround that missing feature would be to use templating and modify the Netlify Functions source files during the build. In fact, there is a plugin for this: netlify-plugin-inline-functions-env.

Since you are developing a plugin, your instruction could ask users to use your plugin together with that other plugin (there is currently no documented pattern for plugins composition).

Please let me know what you think!

dthyresson · July 15, 2020, 3:19pm

Thanks!

I had a look at the netlify-plugin-inline-functions-env plugin as see what it is doing.

Because in my case I want to:

set build time env
to be used in a prisma.schema file

as modified by this specific plugin

And the file to swap out the value is a .schema files and not .js or the functions dir, I cannot use that specific plugin.

But I can either do the same thing, but modify the prisma.schema to inline the value or change the behavior of netlify-plugin-prisma-provider to both inline the provider as well as the env(DATABSE_URL).

Thanks. I’ve got options now and one will likely work.

ehmicky · July 16, 2020, 11:07am

An alternative could also be sending a PR to netlify-plugin-inline-functions-env in order to allow additional file extensions beyond .js.

dthyresson · July 16, 2020, 2:13pm

True - I considered such a PR, but given the name and scope of netlify-plugin-inline-functions-env was about “functions” and the files I need to change are prisma.schema I thought it might be confusing.

Unless the plugin becomes more of a general env template replacer.

I think I am going to add a “prisma” replacement to my heroku plugin instead.

Thanks again.

Sensibleish · July 17, 2020, 4:52am

@ehmicky, question related to your statement that the only environment variables available in Netlify Functions at runtime are the ones specified in the Netlify app UI. If you use the UI to change variable values, when will already-deployed functions get the new values?

dthyresson · July 17, 2020, 10:02am

Oh - that’s an interesting question. Are they set at build time (for functions) or fetched per each function invocation?

ehmicky · July 17, 2020, 10:45am

Each build re-deploys all Netlify Functions, even if their source code has not changed, i.e. I am not sure what you mean by “already-deployed functions”.

And the environment variables set in Netlify app UI during that build, in addition to the ones usually available in any AWS Lambda Function, should be available runtime. So, the sets of available environment variables is determined at build time, not fetched each time the Function is called.

dthyresson · July 17, 2020, 11:21am

If I had to guess what @Sensibleish is asking is:

I set an env in the UI TOKEN=foo
I deploy a function run that logs out the process.env.TOKEN
change the value of TOKEN to moo in UI without redeploying run
when run executes, do I see foo or moo?
I think I still see foo
I rebuild and function deploys again, I know I will see moo.

For example, take Heroku. If you change a config setting the app restarts so the new config takes effect – ie, essentially a “redeploy”.

ehmicky · July 20, 2020, 10:17am

Yes, I can confirm Netlify does work as you describe, i.e. you need to explicitly trigger a new deploy for any changes in the environment variables to be available in Netlify Functions.

In that respect, it does not work “automatically” as Heroku would.

dthyresson · July 20, 2020, 11:32am

Thanks for confirming. Last question I promise.

I’ve looked in the CLI and API and did not see a similar equivalent to Heroku’s

heroku config:set ENV=value

Is this possible via the Netlify CLI/API?

Thanks.

ehmicky · July 21, 2020, 10:37am

Last question I promise.

Please feel free to ask as many questions as you need!

You can achieve this using:

netlify api updateSite --data='{ "site_id": "418b94bc-93cd-411a-937a-ae4c734f17c4", "body": { "build_settings": { "env": { "ENV": "value", "OTHER_ENV": "otherValue" } } }}'

The site_id value must be replaced by the Site ID (you find that ID in the Settings > General > Site Details > Site information > API ID).

This modifies the environment variables set in the Netlify app. This does not modify the build.environment set in your netlify.toml.

One catch though: this replaces all Site’s environment variables at once. So, if you have existing environment variables, those should be set with that command as well, or they would be removed. You can retrieve the current environment variables using a similar command, and looking for the build_settings.env properties:

netlify api getSite --data='{ "site_id": "418b94bc-93cd-411a-937a-ae4c734f17c4" }'

I agree this is a little complicated though. I just submitted a feature request to Netlify CLI to add a proper CLI command for this use case.

dthyresson · July 21, 2020, 12:00pm

That’s what I thought – well actually I thought I’d have to send all the site data attributes up not just the env. And that was going to be cumbersome.

But if it is just the build_settings … I can work with that especially via the API.

What I aim to do is

github.com

dthyresson/netlify-plugin-heroku-database-url/blob/master/src/index.js#L63


      
              try {
                const options = {
                  appName: inputs.app_name,
                  config: inputs.heroku_config || 'DATABASE_URL',
                }
          
                log(
                  `Setting Netlify ENV ${inputs.netlify_env} to Heroku config database connection.`
                )
          
                process.env[inputs.netlify_env] = await getDatabaseUrl({
                  options,
                })
              } catch (error) {
                const message = `Unable to update ${inputs.netlify_env} from Heroku config`
                log(message, ERR)
                log(error, ERR)
                utils.build.failBuild(message)
              }
            },
          }

set the DATABASE_URL env from a value fetched from a Heroku API call.

So, instead, here I can call the Netlify API instead to set.

Thanks.

Yes - having an API “put” that can operate on a single build_settings.env to add, edit, delete would be ideal.

raae · January 22, 2021, 10:56am

However in this blog post Functions are used as the example recipient of env variables swapped by a build plugin: How to Contextualize your env vars with Netlify Build Plugins. Is this not true then?

Scott · January 22, 2021, 11:54am

Hey @raae,

The blog post illustrates that the plugin is capable of contextualising your environment variables at build time, but unfortunately not for functions at runtime.

You’ll still want to contextualise your functions. This could be achieved with a pre-processing step to inline them, perhaps using a rollin up webpack plugin .

raae · January 22, 2021, 1:38pm

Do you have any nice documentation on this? I have a Gatsby app that get the correct Stripe environment, and I need the function that is deployed with it to use the same environment.

Scott · February 1, 2021, 5:39pm

Hiya!

Hmmm. I think the best thing to check would be this: How to use environment variables in Netlify functions

These are only set at build time but an environment variable is definitely available in a function!