Running a script that updates a file before build

I have a Jekyll site on Netlify and would like to be able to have the following workflow, but not sure how (or the simplest way) to accomplish:

  1. Update a CSV file in my repo (commit / push to master)
  2. Have Netlify run an NPM script I’ve created that creates some json files based on that updated csv and saves them to the _data dir (prior to building the site)
  3. Have Netlify build my Jekyll app including the newly created json files

I know how (and have successfully) ran the NPM script prior to Netlify build, but of course the changes to the json files are not committed/pushed to the repo, so the subsequent build does not have them…

So is it possible/recommended that I commit / push that change from within Netlify (there by triggering another build)? Or is there a better solution to this problem?

Shouldn’t be a problem. What have you tried? I’d do something like this, at a high level, as a build command:

  1. we clone your repo, and run bundle install automatically,
  2. then we run your build command:
    node ./make-my-json.js && jekyll build
  3. ???
  4. Profit :slight_smile:
1 Like

Thanks for the response. I’ve tried this:

In my package.json:

    "scripts": {
        "prebuild": "npm run gen-json-records",
        "build": "jekyll build --verbose --incremental",
        "gen-json-records": "node generateJsonRecords.js"
    }

and in my netlify.toml file:

    [build]
      command = "npm run build"

This seems to work (as in the script runs) but the build does not seem to include the updated files for some reason. I was thinking because the changes were not actually committed to the repo.

But perhaps my script is running before Netlify clones the repo so it’s being overwritten by the cloned repo? How can I run a script before jekylll build but after Netlify clones my repo?

Exactly the way I already showed you :slight_smile: Use a command like β€œnode ./make-my-json.js && jekyll build”. That will work without npm scripts; I use it on tons of sites.

You can debug the build locally using the workflow here if you’d rather try it on your computer in our environment:

That will let you β€œsee” what would be deployed to understand what files are where!

Thanks very much! I didn’t realize that what I had was actually working the whole time :man_facepalming:. My test of it working was just flawed.

Anyway, appreciate your help! :grin:

1 Like

So can you also run larger scripts such as:
cd dir && touch newfile.js && echo "module.exports = {some configurations here}" >> newfile.js && cd .. && npm run build ?

Also, how do you handle secret management in Netlify?

1 Like

Hey @sschipke, yep, something like that should work. To get really fancy with it, you may be interested in checking out our new (beta) Build Plugins feature:

As for secrets, I think this is probably a good place to start:

Want to check that out and let us know if you have any other questions?

Is it at all possible to add a command in the netlify.toml file or my build.sh file that allows copying Netlify’s global environment variables into a new .env file? I am using a flutter web project and it requires a .env file in the main directory… But of course I can’t upload the .env file to github. Because it’s not using JavaScript, it does not recognize the environment variables in Netlify because it does not point to process.env, and instead looks for a .env in my repo.

My only option at this point is to take the envioronment variables defined in Netlify and build a new .env file before everything else. Currently I have a build.sh file and a netlify.toml file the points to the build.sh. But I do not know how to access the environment variables from Netlify in the script.

Hi @wfs,

You can acres variables in a script like: $VAR_NAME. Does that work for you?

So I got this to work by doing making a build script and then running that build script in my netlify.toml file.

this is a file i made in my root directory – build_env.js

const fs = require("fs");

const envContent = `
KEY1=${process.env.KEY1}
\n
KEY2=${process.env.KEY2}
\n`;

fs.writeFileSync("./env", envContent);

And then I added it to my netlify.toml file

[build]
  command = "node ./build_env.js && {{YOUR_BUILD_COMMAND}}"

However, I was still getting CORS errors when trying my API calls. I realized it was much better and safer to use Netlify serverless functions instead. Im using Flutter Web and the dotenv package required the .env file in the assets. So I just removed the package that required the .env when building the project and shifted everything over to Netlify serverless functions instead. Works like a charm.

I think that when rebuilding the .env file on deploys, the keys can still be viewed if they are being used in API calls so I just considered it useless even if I got rid of the CORS issues. Flutter web + netlify functions seems to be easiest and only way to use environment variables when deploying with Flutter web

1 Like

Thanks so much for sharing your solution! This will definitely be beneficial for future Forums members who encounter something similar. :netliconfetti:

Great topic. I really like it. I have an issue and I don’t the reason for such.

In my build logs I see:

3:13:41 PM: ───────────────────────────────────
3:13:41 PM: 1. build.command from netlify.toml
3:13:41 PM: ───────────────────────────────────
3:13:41 PM: ​
3:13:41 PM: $ node ./build/updateGoogleReviewRank.js && hugo
3:13:41 PM: Error writing to file!
3:13:41 PM: Start building sites …
3:13:41 PM: hugo v0.85.0-724D5DB5+extended linux/amd64 BuildDate=2021-07-05T10:46:28Z VendorInfo=gohugoio
…
So, netlify builts the site without new data.

My updateGoogleReviewRank.js is here

Hi there, @bublick :wave:

Thanks for reaching out. We will just need a bit more context before we can dive in further.

Please share the following:

  1. a description of the issue you are encountering
  2. your Netlify site that this issue is associated with
  3. your full build log
  4. any debugging steps you have already taken

Thank you!

Thank you for your interest, @hillary .
I’ve just fixed my issue. Basically, I should run
node updateGoogleReviewRank.js && hugo
and in js script, I should create json file on the same level
const dest = './data/googleReviewsData.json'

1 Like

Thanks for coming back and letting us know! Glad you have a fix :netliconfetti:

Hello! I had the same issue and it is now fixed. Now the issue is when deploying:

4:09:12 PM:   1. build.command from netlify.toml                            
4:09:12 PM: ────────────────────────────────────────────────────────────────
4:09:12 PM: ​
4:09:12 PM: $ node functions/mailer.js && npm run build
4:09:13 PM: Server is running on port: 3001
4:38:47 PM: Build exceeded maximum allowed runtime

It fails every time.

P.S. instead of npm run build, I tried react-scripts build (it’s a React app) but the same thing happened.

Hi, @JuanSebastianM. The clue in this line:

4:09:13 PM: Server is running on port: 3001

You are running a web server in the build image. You can do this, but you need to stop the server once the build completes.

What is happening is this:

  • node functions/mailer.js is started
  • then npm run build is run
  • the build system waits for both commands to exit but node functions/mailer.js stay running
  • the running node functions/mailer.js process prevent the build from completing so it times out instead with this:
4:38:47 PM: Build exceeded maximum allowed runtime

Do you need to keep that process running after the build is complete? If not, you can stop the process once the build is over by changing the build command to this:

node functions/mailer.js && npm run build && pkill -9 node

If there are other questions about this, please let us know.

1 Like

As I’m trying to implement Nodemailer with Express to send emails through a contact form, this feature needs that node functions/mailer.js to run.

I tried changing the build command to the one you mentioned and it didn’t work. The deployment still fails.

Then that won’t work on Netlify. Netlify can only serve static files, not run servers. You might want to take a look at Netlify Forms or Netlify Functions to achieve the functionality you need.

1 Like

So, that was the big issue. I switched to Netlify Functions and it is working perfectly now :slight_smile: But there’s a little issue so I’m going to create a new topic as it doesn’t have anything to do with this one.

Thanks a lot!