Saltar al contenido ForoHackers

End-to-end checksums: the disk's ECC does not cover the whole path

Sistemas de archivosIngeniería de fiabilidadSistemas de almacenamiento
#1027 · bitflip_betty · · Denunciar

Disks already have ECC and CRCs. Why would your file system need its own checksums? Because the disk only protects the part of the path inside the disk.

A block can be damaged in places the drive does not cover: a flaky cable, a controller bug, a bit flip in RAM while the data sits in the page cache, a firmware that writes the right data to the wrong place ("misdirected write"). The drive then happily returns what it was given, with a valid internal ECC, and your file is wrong without anyone noticing.

End-to-end checksums solve that by computing the checksum in the file system, at write time, from the data as the application gave it, and verifying it on every read. The checksum is stored apart from the block it protects (in the parent metadata), so a misdirected write cannot carry a valid checksum with it. File systems like ZFS and Btrfs work this way, and a periodic scrub reads everything and verifies it.

What it gives you:

  • Detection of silent corruption (with a good copy elsewhere, repair).
  • A name for the bug: "bit rot" is rarely the platters decaying. It is usually a path problem.

What it does not give you: protection from bad RAM before the checksum is computed. That is what ECC memory is for. Defence in depth, as the cosmic rays prefer.

#1487 · floppy_disk_farmer · · Denunciar

I have seen this with an old SCSI cable that was just slightly off. Every file copied fine, every checksum I took later disagreed. The disk was blameless. The cable got a funeral.

#1488 · cron_job_carl · · Denunciar

Mind the scrub schedule. A checksum nobody verifies is a decoration. Monthly scrub, alert on any error count above zero, and you will hear about the cable before the backups do.

#1489 · mutex_mike · · Denunciar

Same principle as in networking: you cannot rely on a lower layer's guarantee for an end-to-end property. The only check that counts is made by the endpoint that cares.

Inicia sesión o únete para responder.