Skip to the content ForoHackers
Log inJoin

Characterization tests: how to touch code nobody understands

Maintenance and legacy codeQuality assuranceTesting
#1054 · legacy_cobol_whisperer · · Report

You inherit a program that works, nobody knows exactly how, and you have to change it. Writing "correct" tests is impossible because you do not know what correct is. Write characterization tests instead: tests that record what the code does today.

The procedure:

  1. Pick a unit (a function, a batch step, a report).
  2. Feed it a set of inputs, including ugly real-world ones (anonymised copies, of course).
  3. Save the actual outputs as the expected results ("golden files").
  4. From now on, any change in output fails the test and you must decide whether the change was intended.
def test_report_matches_golden():
    out = run_report_batch("fixtures/records_small.dat")
    assert out == Path("golden/report_small.txt").read_text()

Why it works: you are not asserting that the behaviour is right, only that it did not change by accident. Bugs that have been there for twenty years are now documented as bugs, and fixing one becomes a deliberate act with a diff you can read.

Cautions:

  • Non-deterministic parts (dates, run ids, ordering) must be normalised or injected, or the test is flaky.
  • Keep the fixtures small enough to review. A 2 GB golden file is a hostage, not a test.
  • When you find a real bug, add a test for the correct behaviour and leave the characterization in a clearly named place until you decide to change it.

Calm refactoring is boring refactoring. Boring refactoring is how the nightly batch keeps running.

#1586 · tabs_over_spaces_42 · · Report

Review-wise, this is the best thing you can attach to a risky change: "here is the golden output before and after, and here is the diff, all intended". I will approve that kind of PR every time.

#1587 · race_condition_rita · · Report

One more caution: a golden file that is regenerated with a single command gets regenerated when it fails, without reading the diff. Make the update step explicit and review it like code, or the test turns into a rubber stamp.

#1588 · legacy_cobol_whisperer · · Report

Yes, and that is a social problem more than a technical one. The rule on my team was: the golden update goes in its own commit, with a message explaining what changed and why.

#1589 · two_phase_commitment · · Report

In database land the same thing: capture the result set of the important queries before a schema change, and compare after. I once caught a collation difference that sorted accented names differently. Nobody had asked for that.

#1590 · rubber_duck_supervisor · · Report

What would you do with a part of the code whose output depends on the current time? Think about where the clock should come from before you answer.

#1591 · legacy_cobol_whisperer · · Report

Inject it, duck. Pass the date in as a parameter, default to the real clock only at the outermost layer. Then the test runs "on" the same day forever.

Log in or join to reply.