Skip to the content ForoHackers
Log inJoin

My CI cache key was the branch name. Confession and fix.

Build systemsCI/CDDevOps
#1032 · git_blame_myself · · Report

Sharing a mistake so it is one fewer for you.

Our pipeline cached dependencies to speed up builds. I set the cache key to the branch name, thinking "each branch gets its own cache". Looked clever. For two weeks nobody noticed that after a dependency update the CI kept building with the old versions. Tests passed in CI, failed on laptops. The classic "works on CI" bug, in reverse.

The fix is to key the cache on the content of the files that determine the dependencies:

- uses: actions/cache@v4
  with:
    path: ~/.cache/pip
    key: pip-${{ runner.os }}-${{ hashFiles('requirements.lock') }}
    restore-keys: |
      pip-${{ runner.os }}-

When requirements.lock changes, the hash changes and the cache is rebuilt. restore-keys is a fallback prefix that restores the most recent partial cache as a starting point, so even a miss is faster than nothing.

Rules I wrote on the wall:

  1. A cache key must include every input that should invalidate it (lockfile, OS, tool version).
  2. A cache is an optimisation. A build with an empty cache must give the same result. Run one clean build on a schedule to check.
  3. Never cache things that hold secrets or build outputs you ship.

Also, yes, I wrote it, I merged it, and I approved my own change.

#1505 · yak_shaving_daily · · Report

Rule 2 is the one. I have a weekly clean build in my homelab pipeline now, mostly out of fear. It has caught two things already, one of them a fetch from a mirror that no longer existed.

#1506 · race_condition_rita · · Report

Put it as a testable claim: "build output is a pure function of the sources". A clean scheduled build is the test for that claim. If the two builds ever differ, you have found a hidden input.

#1507 · cron_job_carl · · Report

Add an alert on "cache hit rate dropped to zero". It usually means someone changed the key format, and the cost (in minutes) arrives on Monday.

Log in or join to reply.