Can one trigger a rebuild when text at a url changes or an api call changes or database data changes?

I recently added functionality to my website generator Nift (https://nift.cc, it’s three times faster than Hugo so probably the world’s fastest website generator) so that users can run scripts and system calls and input the output of both at any point when building webpages.

This is incredibly useful for things like inputting text from a url using cURL, fetching content using api calls, integrating with databases, etc. etc. and that’s just the tip of the ice berg.

With Nift you can already get it to just build the pages that have updated (whether it be one of the used template files that have changed, an included content file, an included script, page title, template path, etc.) by running nsm build-updated, however I don’t have a way to trigger a rebuild when the output of a script or system call changes, for example when the text at a url changes or the output of an API call changes or database data has changed, etc…

Is there currently any way to do this with platforms like Netlify with webhooks maybe? If it’s currently not possible, it would help evolve JAMstack if anyone were able to figure out a nice way to trigger rebuilds in such situations.

Maybe a cms like service where you can add webhooks to notify a webhook at Netlify would work best, rather than having Netlify frequently fetch url content, api calls, database queries etc. and compare them against stored values.

Ideally you would be able to specify the build command for each url, api call, for specific cases with accessing a database, and for each script/system call so you could for example specify which pages to build in specific situations, or trigger a rebuild of the entire website (even if one can trigger a rebuild in this way, with Nift you would still need to either manually specify which pages to build in each case or build the whole site, build updated would still not know what’s changed and needs rebuilding, trying to figure out a nice way to do that, though it’s going to require storing previous output values for curl/api/database calls and re-running all curl/api/database/script/system calls when checking which pages need building which is expensive!).

You can currently build all pages for a basic 10k page website using Nift in about 1.5 seconds on a 2014 11" i5 macbook air running Ubuntu with gnome-session-flashback as the desktop environment (see https://hugo-vs-nift.gitlab.io/). It can check which pages have been updated and need rebuilding in about 0.4 seconds (and assuming not many pages need rebuilding, build the updated pages in basically 0 seconds). This is compared to taking 4.7 seconds to build the exact same website with Hugo, and who knows how long with the slower website generators that are available. So in some respects Nift can be up to 12 times faster than Hugo for the exact same 10k page website!

As JAMstack becomes more widespread there is going to be more and more use cases for JAMstack websites with thousands or hundreds of thousands of pages, at which point these statistics are going to become REALLY important (some website generators like Nift can be used for more than just static/JAMstack websites too, and can for example sit on a server rebuilding updated pages quite frequently so that pages don’t need to be built every time they’re served even if content is updating regularly, there are times where this is better and other times where it’s better to build every time a page is served). So I’m not sure how keen I am to slow build-updated down, maybe I could add in a new Nift command that runs build-updated but also reruns curl/api/database/script/system calls and compares them against stored versions to see what needs building. However the process of storing the output values of such calls will also slow down the build process, it might then be worth having the option to build without storing curl/api/database/script/system calls but that starts getting a bit hairy with regards to users not knowing what’s going on anymore.

You can create site specific webhooks to trigger a build at Netlify when the webhook is called.

Netlify is designed for JAMStack sites. This means we host static files and all dynamic content is done with javascript and API calls.

I mention this because of the following comment:

Nift can be used for more than just static/JAMstack websites too, and can for example sit on a server rebuilding updated pages quite frequently so that pages don’t need to be built every time they’re served even if content is updating regularly

With Netlify, there is no way to run a service/server or other language runtime on the server itself. So it wouldn’t be possible to run this service at Netlify. You could run it elsewhere and use it to trigger builds at Netlify too using the webhook though.

You can create site specific webhooks to trigger a build at Netlify when the webhook is called.

Does this include for example when an api call changes or the text at a url changes? It’d be awesome if you could do different build commands for different webhooks for the same website, is that possible?

Yep aware that Netlify is for JAMstack/static sites, not expecting users to be able to run Nift like it’s on a server with Netlify, was just mentioning how much more general Nift is :).

You could run it elsewhere and use it to trigger builds at Netlify too using the webhook though.

That’s pretty neat!