AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

property-based-testing

Writes, reviews, and debugs property-based tests — Hypothesis, fast-check, proptest, jqwik, rapid, and Echidna or Medusa for Solidity invariants. Use whenever tests should cover a whole input domain instead of a hand-picked list of examples: encode/decode and serialize/deserialize pairs, parsers, canonicalizers and normalizers, validators, numeric and Decimal types, comparators and sort order, data structures, and smart-contract state invariants. Also use when adding cases to an existing @given, fast-check, or proptest suite, when judging whether existing property tests assert anything real, and when a generator has shrunk a counterexample and you need to tell a wrong property from a genuine bug. Not for coverage-guided binary fuzzing (libFuzzer, AFL), mutation-testing campaigns, static analysis, benchmarking, or end-to-end UI tests.

Original source
trailofbits
License
CC-BY-SA-4.0
Pinned revision
2be6433b39e009cd2b6fd3be1895f6b1faa5af9a

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read trailofbits--property-based-testing at revision 2be6433b39e009cd2b6fd3be1895f6b1faa5af9a, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 7

references/reviewing.md

Reviewing Property-Based Tests

A property test can pass for years while asserting nothing. These are the ways that happens, worst first.

Report every issue you find with its severity attached. Do not decide on the author's behalf that a MEDIUM is not worth mentioning.

| Issue | Severity | How it shows up | |---|---|---| | Tautological | CRITICAL | Assertion is true regardless of the implementation | | Vacuous | CRITICAL | assume() filters out nearly everything, or contradicts itself | | No assertion | HIGH | Body calls the function and stops | | Reimplementation | HIGH | Assertion recomputes the function's own logic | | Weaker property available | MEDIUM | Length checked, ordering not | | Over-filtered | MEDIUM | Stacked assume() where a strategy constraint belongs | | Settings | LOW | max_examples=5, or no deadline on an expensive strategy |

Tautological

@given(st.integers())
def test_useless(x):
    result = compute(x)
    assert result == result

Nothing about compute can make this fail.

But f(x) == f(x) is not automatically tautological. It is a real determinism property whenever f is not obviously pure — serializers over dicts or sets, anything touching iteration order, hashing, or time. pickle.dumps(obj) == pickle.dumps(obj) genuinely fails for objects with a nondeterministic __reduce__. Ask whether a broken implementation could falsify it. If yes, it is a property; if no, it is noise.

Vacuous

@given(st.integers())
def test_vacuous(x):
    assume(x > 100)
    assume(x < 50)
    assert compute(x) > 0

Hypothesis reports this as passing before it eventually errors on exhausted filters — and in CI nobody reads the warning. assume(x == 42) is the subtler version: it runs, it passes, and it is an example test wearing a @given decorator.

Reimplementation

@given(st.integers(), st.integers())
def test_reimplements(a, b):
    assert add(a, b) == a + b

If add is a + b, this asserts a + b == a + b. The test survives any bug the two expressions share. Reach for an algebraic property instead — commutativity, identity, associativity — which constrains the function without restating it.

Finding the tests

rg "@given\(|from hypothesis import" --type py
rg "fc\.(assert|property)" --type ts --type js
rg "proptest!|#\[quickcheck\]" --type rust

What to push for

Compare each test against the property catalog in SKILL.md and name the strongest property the code supports but the suite does not assert. A suite that checks len(sort(xs)) == len(xs) and never checks ordering is the common case.

Also worth flagging: floating-point equality without a tolerance, assertions on dict/set iteration order, and anything reading the clock — these produce flakes that get blamed on Hypothesis and then get deleted.