Error with cache using build plugin: hangs when cache asked to list after store

I’m working on adding caching support via a custom plug-in to the netlify build. I have two builds in a row that have hung at the same place:
https://app.netlify.com/sites/docs-tockosorg/deploys/5ec82cb5a0f1c700071795bc
https://app.netlify.com/sites/docs-tockosorg/deploys/5ec839b6d6e81600076330b7

The hang seems to occur if I try to list the files that have been cached after performing the cacheing. Here’s the important bits of the plug-in:

module.exports = {
	async onPreBuild({ constants, utils }) {
		const cacheDirs = getCacheDirs(constants);

		console.log('Cache directories:', cacheDirs);

		const files = await utils.cache.list();
		console.log('Netlify cache list', files);

		if (await utils.cache.restore(cacheDirs)) {
			console.log('Tock build cache found -- using cache.');
		} else {
			console.warn('No cache found.');
		}
	},

	async onPostBuild({ constants, utils }) {
		const cacheDirs = getCacheDirs(constants);

		console.log('Cache directories:', cacheDirs);

		if (await utils.cache.save(cacheDirs)) {
			console.log('Stored Tock build cache.');
		} else {
			console.error('Failed to save cache.');
		}

		const files = await utils.cache.list();
		console.log('Netlify cache list', files);
	},
}

The “print cache” logic works correctly before the attempt to restore in the PreBuild, but hangs in the PostBuild:

4:45:12 PM: Cache directories: [ '/opt/buildhome/.cargo', 'target' ]
4:45:12 PM: Netlify cache list []
4:45:12 PM: No cache found.

...

4:46:53 PM: Cache directories: [ '/opt/buildhome/.cargo', 'target' ]
4:47:00 PM: Stored Tock build cache.
<hang>

The full plug-in code that is running is in this PR: https://github.com/tock/tock/pull/1875/files

by plug in do you mean Build Plugins? Just want to clarify.

Yep!

We’ve opted in to the beta rollout and are using an “unpublished”, local to our repo plugin (also listed in the PR linked in the first comment)

Hi @ppannuto,

Do you know how many files are inside those two cache directories?

There is a chance utils.cache.list() is taking a long time to complete if the number of files to list is very high. You can check the code for this method here: all it is doing essentially is using readdirp, which performs fs.readdir() recursively.

I have also opened the following issue to make utils.cache.list() incremental instead of buffering all results at once.

Hm, it’s not the smallest number of files. From a coarse analysis, I’d say the target/ folder has around 8k files and the .cargo directory closer to 25k.

Is there a limit to the number of files that can be cached? I added the list to start debugging why the cache was empty every build (and I wanted to prove that anything was put in the cache).

I could try tar’ing the folders so that only a single file is cached?

There is not a hard limit to the number of files that can be cached, except for what the OS (Ubuntu on Docker) can handle.

The solution to this problem would be to improve the utils.cache.list() utility following your suggestion on this issue. I added a potential solution here.