Property-Based Testing

By Michael Samuel Naeem · January 6, 2026 · 8 min read

TestingProperty-Based TestingQualityEngineering

Our sort function passed all twelve unit tests. A property-based test found the failure on its third generated input: sort([3, 3, 1]) returned [1, 3] — duplicate elements were silently dropped. The unit tests used unique elements exclusively.

Property-based testing verifies that certain properties hold for all inputs, not just the handful you thought to test. The framework generates random inputs, runs your property function, and when a failure occurs, shrinks the input to the smallest failing case.

Properties vs examples

Example-based: "sort([3, 1, 2]) returns [1, 2, 3]" — one case.

Property-based: "for any list, sort output is ordered, same length, and contains the same elements" — all cases.

from hypothesis import given, strategies as st

@given(st.lists(st.integers()))
def test_sort_preserves_length(input_list):
    assert len(sort(input_list)) == len(input_list)

@given(st.lists(st.integers()))
def test_sort_is_ordered(input_list):
    result = sort(input_list)
    assert all(result[i] <= result[i+1] for i in range(len(result)-1))

Hypothesis (Python)

@given(st.integers(), st.integers())
def test_addition_commutative(a, b):
    assert add(a, b) == add(b, a)

Shrinking: When a test fails, Hypothesis simplifies the input to the minimal failing case — easier to debug than a 100-element random list.

Common properties to test

Roundtrip: decode(encode(x)) == x

Idempotency: sort(sort(lst)) == sort(lst)

Inverse operations: decode(encode(n)) == n

Postconditions: After remove(lst, item), item is gone and length decreased by one.

jqwik (Java/Kotlin)

@Property
void sortPreservesLength(@ForAll List<Integer> input) {
    assertThat(Sorter.sort(input)).hasSize(input.size());
}

Custom generators

Build domain-valid inputs when built-in strategies aren't useful:

@st.composite
def valid_orders(draw):
    items = draw(st.lists(st.fixed_dictionaries({
        "sku": st.text(min_size=1), "qty": st.integers(min_value=1, max_value=100),
    }), min_size=1))
    return {"items": items}

Combining with example tests

Examples document expected behavior. Properties verify invariants hold broadly. Both in the same test file.

Operational checklist for teams

Assign an owner for each recurring failure mode. Measure baseline before changes and compare after two sprints. Document the fix in the team wiki or runbook so the next engineer does not rediscover it. Schedule a quarterly review of metrics, tooling, and process — what reduced flakes or improved quality last quarter should be doubled down on; what did not work should be retired explicitly rather than left on the books.

Operational checklist for teams

Stateful testing for complex domains

When properties involve sequences of operations — push/pop, credit/debit, open/close — use stateful property testing (Hypothesis RuleBasedStateMachine, QuickCheck dynamic). Define rules for valid operations and invariants that must hold after every step. Model-based testing compares a simple reference implementation against optimized production code — if they diverge on random operation sequences, the optimized version has a bug.

Shrinking readable counterexamples

Hypothesis shrinking produces minimal failing inputs — teach devs to read shrunk output, not re-run with fixed seed only. Store @example decorators for every shrunk case in regression suite so CI catches reintroduction without full property search.

Shrinking noise in CI

Cap Hypothesis max_examples at 100 in CI, 1000 locally — full search runs nightly. @settings(deadline=None) on slow properties to avoid flaky timeout on loaded CI runners.