I have a Jekyll site on Netlify and would like to be able to have the following workflow, but not sure how (or the simplest way) to accomplish:
Update a CSV file in my repo (commit / push to master)
Have Netlify run an NPM script Iβve created that creates some json files based on that updated csv and saves them to the _data dir (prior to building the site)
Have Netlify build my Jekyll app including the newly created json files
I know how (and have successfully) ran the NPM script prior to Netlify build, but of course the changes to the json files are not committed/pushed to the repo, so the subsequent build does not have themβ¦
So is it possible/recommended that I commit / push that change from within Netlify (there by triggering another build)? Or is there a better solution to this problem?
This seems to work (as in the script runs) but the build does not seem to include the updated files for some reason. I was thinking because the changes were not actually committed to the repo.
But perhaps my script is running before Netlify clones the repo so itβs being overwritten by the cloned repo? How can I run a script before jekylll build but after Netlify clones my repo?
Exactly the way I already showed you Use a command like βnode ./make-my-json.js && jekyll buildβ. That will work without npm scripts; I use it on tons of sites.
You can debug the build locally using the workflow here if youβd rather try it on your computer in our environment:
That will let you βseeβ what would be deployed to understand what files are where!
So can you also run larger scripts such as: cd dir && touch newfile.js && echo "module.exports = {some configurations here}" >> newfile.js && cd .. && npm run build ?
Also, how do you handle secret management in Netlify?
Hey @sschipke, yep, something like that should work. To get really fancy with it, you may be interested in checking out our new (beta) Build Plugins feature:
As for secrets, I think this is probably a good place to start:
Want to check that out and let us know if you have any other questions?
Is it at all possible to add a command in the netlify.toml file or my build.sh file that allows copying Netlifyβs global environment variables into a new .env file? I am using a flutter web project and it requires a .env file in the main directoryβ¦ But of course I canβt upload the .env file to github. Because itβs not using JavaScript, it does not recognize the environment variables in Netlify because it does not point to process.env, and instead looks for a .env in my repo.
My only option at this point is to take the envioronment variables defined in Netlify and build a new .env file before everything else. Currently I have a build.sh file and a netlify.toml file the points to the build.sh. But I do not know how to access the environment variables from Netlify in the script.
However, I was still getting CORS errors when trying my API calls. I realized it was much better and safer to use Netlify serverless functions instead. Im using Flutter Web and the dotenv package required the .env file in the assets. So I just removed the package that required the .env when building the project and shifted everything over to Netlify serverless functions instead. Works like a charm.
I think that when rebuilding the .env file on deploys, the keys can still be viewed if they are being used in API calls so I just considered it useless even if I got rid of the CORS issues. Flutter web + netlify functions seems to be easiest and only way to use environment variables when deploying with Flutter web
Great topic. I really like it. I have an issue and I donβt the reason for such.
In my build logs I see:
3:13:41 PM: βββββββββββββββββββββββββββββββββββ
3:13:41 PM: 1. build.command from netlify.toml
3:13:41 PM: βββββββββββββββββββββββββββββββββββ
3:13:41 PM: β
3:13:41 PM: $ node ./build/updateGoogleReviewRank.js && hugo
3:13:41 PM: Error writing to file!
3:13:41 PM: Start building sites β¦
3:13:41 PM: hugo v0.85.0-724D5DB5+extended linux/amd64 BuildDate=2021-07-05T10:46:28Z VendorInfo=gohugoio
β¦
So, netlify builts the site without new data.
Thank you for your interest, @hillary .
Iβve just fixed my issue. Basically, I should run node updateGoogleReviewRank.js && hugo
and in js script, I should create json file on the same level const dest = './data/googleReviewsData.json'
You are running a web server in the build image. You can do this, but you need to stop the server once the build completes.
What is happening is this:
node functions/mailer.js is started
then npm run build is run
the build system waits for both commands to exit but node functions/mailer.js stay running
the running node functions/mailer.js process prevent the build from completing so it times out instead with this:
4:38:47 PM: Build exceeded maximum allowed runtime
Do you need to keep that process running after the build is complete? If not, you can stop the process once the build is over by changing the build command to this:
node functions/mailer.js && npm run build && pkill -9 node
If there are other questions about this, please let us know.
Then that wonβt work on Netlify. Netlify can only serve static files, not run servers. You might want to take a look at Netlify Forms or Netlify Functions to achieve the functionality you need.
So, that was the big issue. I switched to Netlify Functions and it is working perfectly now But thereβs a little issue so Iβm going to create a new topic as it doesnβt have anything to do with this one.