Deterministic Tests
Same code, same result, every run. What this property actually protects is trust in the suite as a whole: one flaky test teaches the team to rerun on red, and from then on the suite’s verdict means nothing. “Once you start ignoring a regression test failure, then that test is useless and you might as well throw it away” (Fowler).
Terminology worth keeping straight: Beck, Google, and GOOS say deterministic; Meszaros and FIRST say repeatable. Reading those as a strict-to-permissive spectrum is this note’s lens, not a line the sources draw. At the strict end: no randomness at all. At the permissive end, a seeded RNG, fixed clock, or stubbed network all qualify. Beck’s own definition sits at the permissive end: “if nothing changes, the test result shouldn’t change”, which a seeded RNG or fixed clock satisfies. The Google book likewise just asks for deterministic with no no-randomness gloss. Repeatable is the bar that matters: no flakes.
The clock, network, and concurrency are both the usual flake sources and part of how software really runs, so stubbing them trades against Behavioral fidelity, one of the trade-offs among the properties.
Part of Test Desiderata.
Measuring: per-test flakiness rate and rerun-minutes as a share of CI time, with a quarantine for repeat offenders. The block-on-flake threshold is a team policy call, not a universal number; what matters is that flakiness is common enough to need an explicit one.
Sources
- Martin Fowler, Eradicating Non-Determinism in Tests (2011): an ignored test is useless, and a handful of non-deterministic tests destroy trust in the whole suite.
- Gerard Meszaros, Goals of Test Automation: the Repeatable Test, the same no-flakes bar under a different name.
- Freeman and Pryce, Growing Object-Oriented Software, and Software Engineering at Google: both call the property deterministic: the stricter-sounding label for the same no-flakes bar, not a stricter bar.