Skip to the content ForoHackers
Log inJoin

Storage systems: start here

Storage systems
#425 · Ophuscado · · Report

Storage systems keep data safe, reachable and fast, from one disk to thousands across data centres. Every design trades capacity, speed, durability and cost, and the interesting engineering is in surviving failure: disks die, power drops mid-write, and corruption can sit unnoticed for years.

Key concepts

  • Storage models: block devices, files and objects, each with its own interface and sweet spot.
  • Media hierarchy: memory, flash and spinning disks differ by orders of magnitude in latency and cost.
  • Redundancy: mirroring, RAID and erasure coding trade extra space for tolerance of failed disks.
  • Replication and consistency: copies across machines or sites, and what a reader sees after a write.
  • Durability: write-ahead logs, flushing to stable media, checksums and scrubbing to catch silent corruption.
  • Data layout: log-structured designs, caching and tiering shape performance more than raw hardware does.

Vocabulary

  • IOPS and throughput: operations per second and bytes per second, two different limits.
  • RAID: combining disks for speed, redundancy or both.
  • Erasure coding: splitting data into pieces so it can be rebuilt from a subset.
  • Snapshot: a frozen, consistent view of data at a point in time.
  • Fsync: the call that forces data to stable storage.

What to study, in order

  1. How file systems and block devices work on your own machine.
  2. Measure a disk with a benchmarking tool and see random against sequential, and small against large.
  3. Build a RAID or mirror from files in a lab, fail a member and rebuild it.
  4. Write a small log-structured key-value store with a write-ahead log and crash-test it.
  5. Study replication and erasure coding in a distributed store, and compute the space cost of each.

Related tags

#file-systems #databases #backups #distributed-systems #performance-evaluation

Log in or join to reply.