Diagnosable Tests

Bazuzi’s name for Beck’s Specific, adopted because it states the goal: when a test fails, the cause should be obvious from the failure alone. The ideal is Bazuzi’s “1 defect == 1 test failure”: a single broken behavior lights up exactly one test, and its message carries the offending input.

Eager tests (one test asserting many behaviors) and assertion roulette (which assert fired?) are the classic smells, first named by van Deursen and colleagues in 2001 and catalogued by Meszaros (Eager Test, Assertion Roulette). Diagnosability leans on readability and on order independence; a cascade of collateral failures diagnoses nothing. It pulls the other way against Structure-Insensitive, though: a test that ignores internals fails over a broad surface and localizes the defect worse, one of the trade-offs among the properties.

Part of Test Desiderata.

Measuring: mean time from red CI to fix commit, assertion-message completeness, and mutants killed per test (a test killing many unrelated mutants is doing too much).

Sources

• Jay Bazuzi, Characteristics of Ideal Tests: “1 defect == 1 test failure.”
• Arie van Deursen, Leon Moonen, Alex van den Bergh, Gerard Kok, Refactoring Test Code (XP2001): first named Eager Test and Assertion Roulette.
• Gerard Meszaros, Obscure Test: the Eager Test cause, and Assertion Roulette, which is its own behavior smell.