Processing images Gulp and building time

Support
1

Hello, everyone! I have built a site using Hugo + Gulp, attached NetlifyCMS and moved to Netlify.
This site contains about 100 images and they are all processed with gulp-responsive to save space and add webp support. Everything works great, but small but:
Each time I do even small text change using NetlifyCMS or locally, the build script processing all images I have and increasing build time much. Locally using gulp-newer helps me, but on netlify servers such feature doesn’t work. I’ve read that there is undocumented folder in netlify “/opt/build/cache/”, but it haven’t helped me a lot. Maybe I missing something. Any advice would be appreciated. Thanks a lot for your time!

gulp.task('images-prod', () => {
 return gulp.src('/opt/build/cache/assets/images/*.{jpg,jpeg,png}')
.pipe(newer('/assets/images/public'))
.pipe(responsive({
  '**/*.{jpg,png,jpeg}': [{
    width: 2000,
    quality: 75,
    compressionLevel: 7,
  }, {
    width: 2000,
    quality: 75,
    rename: {
      extname: '.webp',
    },
  }],
}, {
  errorOnEnlargement: false,
  errorOnUnusedConfig: false
}))
.pipe(rename(function(opt) {
  opt.basename = opt.basename.split(' ').join('_');
  return opt;
}))
.pipe(gulp.dest('/assets/images/public/'));
});

gulp.task('hugo-build', shell.task(['hugo']))

gulp.task('build', gulp.series('minify-js', 'hugo-build', 'images-prod', 'minify-html'));
2

Hi @mncandsons and welcome to our community! I don’t know that this would work - we only cache dependencies by default, so your images won’t be there. You might try using an npm package like this:

which you can configure to save things at the end of build for reuse next time. We can’t advise in more detail, but others use that successfully for similar purposes :slight_smile:

You’ll need to make sure that your build works WITH or WITHOUT the cache (regenerating the cache in case it is not present) since there will be rare times it is unavailable and if your build requires it, it won’t ever get generated/saved at the end of a failing build.

3

hi, @fool. Thanks for your time, spent on this and advice. I’ve started using this package and it seems work well on local machine, but when I am trying to upload it to Netlify logs tell me, that

No ‘cache.json’ found in /opt/build/cache/static/assets

And starts build from scratch. What I do wrong? Wrong path or?

Appreciated for your help!

4

@mncandsons, that cache won’t exist until it is created for the first time. The first step will be storing the required files to a cache directory so it can be restored for use next time. It sounds like the cache isn’t being prepared.

Would you be willing to send us a link to the deploy where this didn’t work? Also, would you mind sharing the code you used to store and retrieve the files to and from this build cache directory?

5

Thank you, @luke . Here will be part of my gulpfile and logs. They are quite long, so I will remove just image processing to save space :
This is a log of build with cleared cache (initial):

2:23:11 PM: Build ready to start
2:23:15 PM: build-image version: 9cade8af58c2cf3a17a1e9433d2e979149488837
2:23:15 PM: build-image tag: v3.3.5
2:23:15 PM: buildbot version: 2fd8f04962d5dbc77b79b9c2f5ab7a077e87376b
2:23:15 PM: Fetching cached dependencies
2:23:15 PM: Starting to download cache of 254.9KB
2:23:15 PM: Finished downloading cache in 80.9828ms
2:23:15 PM: Starting to extract cache
2:23:15 PM: Failed to fetch cache, continuing with build
2:23:15 PM: Starting to prepare the repo for build
2:23:16 PM: No cached dependencies found. Cloning fresh repo
2:23:16 PM: git clone GitHub - mncandsons/mnc
2:23:23 PM: Preparing Git Reference refs/heads/master
2:23:24 PM: Starting build script
2:23:24 PM: Installing dependencies
2:23:26 PM: Downloading and installing node v10.19.0…
2:23:26 PM: Downloading https://nodejs.org/dist/v10.19.0/node-v10.19.0-linux-x64.tar.xz…
2:23:26 PM:
################### 27.3%
2:23:26 PM:
########################################################
2:23:26 PM: ################ 100.0%
2:23:26 PM: Computing checksum with sha256sum
2:23:26 PM: Checksums matched!
2:23:29 PM: Now using node v10.19.0 (npm v6.13.4)
2:23:29 PM: Attempting ruby version 2.6.2, read from environment
2:23:31 PM: Using ruby version 2.6.2
2:23:31 PM: Using PHP version 5.6
2:23:31 PM: Started restoring cached node modules
2:23:31 PM: Finished restoring cached node modules
2:23:31 PM: Installing NPM modules using NPM version 6.13.4
2:23:38 PM: > sharp@0.23.4 install /opt/build/repo/node_modules/sharp
2:23:38 PM: > (node install/libvips && node install/dll-copy && prebuild-install) || (node-gyp rebuild && node install/dll-copy)
2:23:38 PM: info
2:23:38 PM: sharp
2:23:38 PM: Downloading https://github.com/lovell/sharp-libvips/releases/download/v8.8.1/libvips-8.8.1-linux-x64.tar.gz
2:23:40 PM: npm WARN
2:23:40 PM: MNC@1.0.0 No repository field.
2:23:40 PM: npm WARN
2:23:40 PM: optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.2.11 (node_modules/fsevents):
2:23:40 PM: npm WARN
2:23:40 PM: notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.11: wanted {“os”:“darwin”,“arch”:“any”} (current: {“os”:“linux”,“arch”:“x64”})
2:23:40 PM: added 439 packages from 343 contributors and audited 6955 packages in 8.555s
2:23:41 PM: 4 packages are looking for funding
2:23:41 PM: run npm fund for details
2:23:41 PM: found 0 vulnerabilities
2:23:41 PM: NPM modules installed
2:23:41 PM: Started restoring cached go cache
2:23:41 PM: Finished restoring cached go cache
2:23:41 PM: unset GOOS;
2:23:41 PM: unset GOARCH;
2:23:41 PM: export GOROOT=‘/opt/buildhome/.gimme/versions/go1.12.linux.amd64’;
2:23:41 PM: export PATH=“/opt/buildhome/.gimme/versions/go1.12.linux.amd64/bin:${PATH}”;
2:23:41 PM: go version >&2;
2:23:41 PM: export GIMME_ENV=‘/opt/buildhome/.gimme/env/go1.12.linux.amd64.env’;
2:23:41 PM: go version go1.12 linux/amd64
2:23:41 PM: Installing missing commands
2:23:41 PM: Verify run directory
2:23:41 PM: Executing user command: gulp build
2:23:42 PM: [12:23:42]
2:23:42 PM: Using gulpfile /opt/build/repo/gulpfile.js
2:23:42 PM: [12:23:42]
2:23:42 PM: Starting ‘build’…
2:23:42 PM: [12:23:42]
2:23:42 PM: Starting ‘minify-js’…
2:23:43 PM: [12:23:43]
2:23:43 PM: Finished ‘minify-js’ after 1.22 s
2:23:43 PM: [12:23:43]
2:23:43 PM: Starting ‘images-prod’…
2:23:43 PM: No ‘cache.json’ found in /opt/build/cache/storage/opt/build/repo/static/assets
2:23:43 PM: 1. :mag: Checking for changes in /opt/build/repo/static/assets/images …
2:23:43 PM: 2. ✎ Changes detected in /opt/build/repo/static/assets/images. Running cache update logic
2:23:43 PM: 3. ◉ Running handleCacheUpdate function…
2:23:43 PM: Start
2:23:44 PM: [12:23:44]
2:23:44 PM: gulp-responsive: 19.050.000A.jpg → 19.050.000A.jpg
2:23:44 PM: [12:23:44]
2:23:44 PM: gulp-responsive: 19.050.000A.jpg → 19.050.000A.webp
2:23:45 PM: [12:23:45]
2:23:45 PM: gulp-responsive: 19.050.001A.jpg → 19.050.001A.jpg
2:23:45 PM: [12:23:45]
2:23:45 PM: gulp-responsive: 19.050.001A.jpg → 19.050.001A.webp
2:23:46 PM: [12:23:46]
…
2:26:52 PM: gulp-responsive: logo.png → logo.webp
2:26:52 PM: [12:26:52]
2:26:52 PM: gulp-responsive: Created 422 images (matched 211 of 211 images)
2:26:52 PM: 4. Copying over /opt/build/repo/static/assets/images to /opt/build/cache/storage/opt/build/repo/static/assets/images…
2:26:53 PM: 5. ✓ Dependencies copied to cache /opt/build/cache/storage/opt/build/repo/static/assets/images
2:26:53 PM: 6. Saving cache info…
2:27:08 PM: ✓ Dependencies cached for next run!
2:27:08 PM: Cache Location: /opt/build/cache/storage/opt/build/repo/static/assets/images
2:27:08 PM: ====== Netlify cache restored! ======
2:27:08 PM: /opt/build/cache/storage/opt/build/repo/static/assets
2:27:08 PM: [12:27:08] Finished ‘images-prod’ after 3.42 min
2:27:08 PM: [12:27:08]
2:27:08 PM: Starting ‘hugo-build’…
2:27:09 PM: Building sites …
2:27:10 PM: | EN
2:27:10 PM: ±-----------------±----+
2:27:10 PM: Pages | 8
2:27:10 PM: Paginator pages | 0
2:27:10 PM: Non-page files | 0
2:27:10 PM: Static files | 656
2:27:10 PM: Processed images | 0
2:27:10 PM: Aliases | 0
2:27:10 PM: Sitemaps | 1
2:27:10 PM: Cleaned | 0
2:27:10 PM: Total in 1232 ms
2:27:10 PM: [12:27:10]
2:27:10 PM: Finished ‘hugo-build’ after 1.38 s
2:27:10 PM: [12:27:10]
2:27:10 PM: Starting ‘minify-html’…
2:27:10 PM: [12:27:10]
2:27:10 PM: Finished ‘minify-html’ after 339 ms
2:27:10 PM: [12:27:10] Finished ‘build’ after 3.47 min
2:27:10 PM: Skipping functions preparation step: no functions directory set
2:27:10 PM: Caching artifacts
2:27:10 PM: Started saving node modules
2:27:10 PM: Finished saving node modules
2:27:10 PM: Started saving pip cache
2:27:10 PM: Finished saving pip cache
2:27:10 PM: Started saving emacs cask dependencies
2:27:10 PM: Finished saving emacs cask dependencies
2:27:10 PM: Started saving maven dependencies
2:27:11 PM: Finished saving maven dependencies
2:27:11 PM: Started saving boot dependencies
2:27:11 PM: Finished saving boot dependencies
2:27:11 PM: Started saving go dependencies
2:27:11 PM: Finished saving go dependencies
2:27:13 PM: Build script success
2:27:13 PM: Starting to deploy site from ‘public/’
2:27:14 PM: Creating deploy tree asynchronously
2:27:14 PM: Starting post processing
2:27:14 PM: Post processing done
2:27:15 PM: Site is live
2:27:16 PM: 0 new files to upload
2:27:16 PM: 0 new functions to upload
2:28:22 PM: Finished processing build request in 5m6.814752849s

This is a build after (Not complete, to save the space):

2:27:57 PM: Build ready to start
2:27:59 PM: build-image version: 9cade8af58c2cf3a17a1e9433d2e979149488837
2:27:59 PM: build-image tag: v3.3.5
2:27:59 PM: buildbot version: 2fd8f04962d5dbc77b79b9c2f5ab7a077e87376b
2:27:59 PM: Fetching cached dependencies
2:27:59 PM: Starting to download cache of 711.8MB
2:28:18 PM: Finished downloading cache in 19.197918339s
2:28:18 PM: Starting to extract cache
2:28:27 PM: Finished extracting cache in 9.051386215s
2:28:29 PM: Finished fetching cache in 29.94236861s
2:28:29 PM: Starting to prepare the repo for build
2:28:30 PM: Preparing Git Reference refs/heads/master
2:28:31 PM: Starting build script
2:28:31 PM: Installing dependencies
2:28:32 PM: Started restoring cached node version
2:28:35 PM: Finished restoring cached node version
2:28:35 PM: v10.19.0 is already installed.
2:28:36 PM: Now using node v10.19.0 (npm v6.13.4)
2:28:36 PM: Attempting ruby version 2.6.2, read from environment
2:28:38 PM: Using ruby version 2.6.2
2:28:38 PM: Using PHP version 5.6
2:28:38 PM: Started restoring cached node modules
2:28:38 PM: Finished restoring cached node modules
2:28:39 PM: Started restoring cached go cache
2:28:39 PM: Finished restoring cached go cache
2:28:39 PM: unset GOOS;
2:28:39 PM: unset GOARCH;
2:28:39 PM: export GOROOT=‘/opt/buildhome/.gimme/versions/go1.12.linux.amd64’;
2:28:39 PM: export PATH=“/opt/buildhome/.gimme/versions/go1.12.linux.amd64/bin:${PATH}”;
2:28:39 PM: go version >&2;
2:28:39 PM: export GIMME_ENV=‘/opt/buildhome/.gimme/env/go1.12.linux.amd64.env’;
2:28:39 PM: go version go1.12 linux/amd64
2:28:39 PM: Installing missing commands
2:28:39 PM: Verify run directory
2:28:39 PM: Executing user command: gulp build
2:28:39 PM: [12:28:39]
2:28:39 PM: Using gulpfile /opt/build/repo/gulpfile.js
2:28:39 PM: [12:28:39]
2:28:39 PM: Starting ‘build’…
2:28:39 PM: [12:28:39]
2:28:39 PM: Starting ‘minify-js’…
2:28:41 PM: [12:28:41]
2:28:41 PM: Finished ‘minify-js’ after 1.32 s
2:28:41 PM: [12:28:41]
2:28:41 PM: Starting ‘images-prod’…
2:28:41 PM: No ‘cache.json’ found in /opt/build/cache/storage/opt/build/repo/static/assets
2:28:41 PM: 1. :mag: Checking for changes in /opt/build/repo/static/assets/images …
2:28:41 PM: 2. ✎ Changes detected in /opt/build/repo/static/assets/images. Running cache update logic
2:28:41 PM: 3. ◉ Running handleCacheUpdate function…
2:28:41 PM: Start
2:28:42 PM: [12:28:42]
2:28:42 PM: gulp-responsive: 19.050.000A.jpg → 19.050.000A.jpg
2:28:42 PM: [12:28:42]
2:28:42 PM: gulp-responsive: 19.050.000A.jpg → 19.050.000A.webp
2:28:43 PM: [12:28:43]

The Gulp task:

gulp.task(‘images-prod’, (done) => {
const cacheFolder = path.join(‘/opt/build/cache’, ‘storage’);
const contentsToCache = [{
contents: path.join(__dirname, ‘static/assets/images’),
handleCacheUpdate: () => {
console.log(‘Start’);
return new Promise((resolve, reject) => {
gulp.src(‘static/assets/images/.{jpg,jpeg,png}')
.pipe(newer(cacheFolder))
.pipe(responsive({
'**/.{jpg,png,jpeg}’: [{
width: 2000,
quality: 75,
compressionLevel: 7,
}, {
width: 2000,
quality: 75,
rename: {
extname: ‘.webp’,
},
}],
}, {
errorOnEnlargement: false,
errorOnUnusedConfig: false
}))
.pipe(rename(function (opt) {
opt.basename = opt.basename.split(’ ‘).join(’_‘);
return opt;
}))
.pipe(gulp.dest(‘static/assets/images/public’)).on(‘end’, resolve);
});
}
}]
cacheMeOutside(cacheFolder, contentsToCache).then(cacheInfo => {
console.log(’====== Netlify cache restored! ======')
cacheInfo.forEach(info => {
console.log(info.cacheDir)
})
done();
})
});

6

Based on your logs, it looks like you get the following error in both cases you shared: No ‘cache.json’ found in /opt/build/cache/storage/opt/build/repo/static/assets. That path looks a little funky to me, like you’re joining two dirs together instead of maybe replacing one with the other. I’m guessing that happens in your Gruntfile, though I can’t say exactly where. Do you want to troubleshoot that and see if it fixes your build?