Maintaining bitbake sstate-cache in CI

Today we ran out of space on our 500GB cache disk that holds bitbake sstate-cache, bitbake downloads directory and repo git mirrors. This was bound to happen, as I didn’t set up any retention policy when we started using this system several months ago.

We kind of bought more time when we realized we need private/public caches and we purged existing cache to make sure we are not accidentally distributing anything that we may not have permission to but that was a temporary respite.

I’ve set up a naive scheduled job that runs once a month. This was added in .gitlab-ci.yml: allow purging sstate-cache (!15) · Merge requests · OSTC / infrastructure / pipelines · GitLab and there’s a separate schedule entry that is not expressed in git at Pipeline Schedules · OSTC / infrastructure / pipelines · GitLab

I’d like to start this thread to see how we can do this better. Can we remove oldest entries without ruining hot cache? Should we handle cache for tagged builds separately, so that releases stay “hot” (though arguably, I would think that it should be not tags but maintenance branches that have this property as a snapshots are not that interesting in practice).

Perhaps @agherzan has some ideas on how this can be done. I have not yet looked at what bitbake APIs expose. I suspect removing oldest files without maintaining some properties that bitbake relies on is not the best idea.

EDIT: I’ve filed an incident for this outage: Out of disk space on shared NFS volume (#6) · Issues · OSTC / infrastructure / pipelines · GitLab

There are two ways I’d go about this:

  1. A simple option doing something like:
 - du -h -s $SSTATE_DIR
  - find $SSTATE_DIR -type f -atime +30 -delete
  - du -h -s $SSTATE_DIR

In a regular job.

  1. We can be a bit smarter while using something like sstate-cache-management.sh « scripts - poky - Poky Build Tool and Metadata (or even this script directly as a starting point)
1 Like