Plumbdev07-2 · check the tests bite
A sub-flow of dev07 · run plumb. Entered when the project asks for mutation, carrying the trace, the annotations and the story. Exits to dev07-3 · collect what the run did, carrying a score per story — or nothing at all, which is an answer and never a zero.
dev07-2-1 scope to the code the tests drove under an entry point Proven
As a developer reading a mutation score I want it to be about the code this story's tests actually wired, so that the number distinguishes my stories.
The tests that cite it (5)
tests/test_end_to_end.py::test_mutation_separates_a_test_that_checks_from_one_that_merely_runs— passed, groundedtests/test_mutation.py::test_only_code_that_ran_under_an_entry_point_is_mutated— passed, ungroundedtests/test_mutation.py::test_the_scope_is_the_union_over_every_citing_test_grouped_by_file— passed, ungroundedtests/test_mutation.py::test_only_the_named_functions_are_mutated— passed, ungroundedtests/test_the_jvm_adapter.py::test_mutation_scopes_to_what_ran_under_an_entry_point_and_no_wider— skipped, not-checked
Requirements (8)
What the user needs
- dev07-2-1-ur1only code the story's tests drove under a declared entry point is mutated, never the whole call tree
- dev07-2-1-ur2the scope is the union over every citing test, named as a trace names a frame
- dev07-2-1-ur3only the story's passing citing tests enter the plan
- dev07-2-1-ur4a frame the declared source roots cannot place in source is not a target — a project that declares nothing gets no mutants, never a guess
- dev07-2-1-ur5the plan runs only with both tests and mutants in it; otherwise the pass reports nothing at all, because a score out of zero reads like an answer
What the system does
- dev07-2-1-sr1the scope is recomputed from the trace every run; no story-to-code map is written down
- dev07-2-1-sr2targets are placed in the declared source roots, which are declared apart from production because mutation changes source, and a compiled language does not run its source
Still open
- dev07-2-1-u1narrowing further to only what changed needs a definite diff, and the mechanism is the adapter's to fix
dev07-2-2 price the pass before it runs Proven
As a developer about to start this I want the price before I pay it, so that deciding whether it is worth an hour is a decision I can make.
The tests that cite it (5)
tests/test_end_to_end.py::test_mutation_separates_a_test_that_checks_from_one_that_merely_runs— passed, groundedtests/test_end_to_end.py::test_mutation_progress_counts_what_it_mutates_and_starts_on_its_own_line— passed, groundedtests/test_mutation.py::test_the_cost_can_be_read_off_before_any_of_it_is_spent— passed, ungroundedtests/test_mutation.py::test_mutation_is_off_unless_the_project_asks_for_it— passed, ungroundedtests/test_mutation.py::test_with_the_opt_in_off_the_slot_is_left_unfilled_and_nothing_runs— passed, ungrounded
Requirements (5)
What the user needs
- dev07-2-2-ur1the price is reported before any mutant runs
- dev07-2-2-ur2the price is per story, so a caller picks a subject they can afford
- dev07-2-2-ur3the price is a floor and says so, and is revised from the first completed mutant — while stopping is still free
What the system does
- dev07-2-2-sr1the count comes from one parse, never from generating the mutants
- dev07-2-2-sr2the dominant term is what the story's citing tests cost earlier in this same run — measured moments before, never assumed
dev07-2-3 run every citing test against each mutant Proven
As a developer whose story has four tests I want all four run against a mutant, so that unnoticed means no test of this story caught it.
The tests that cite it (10)
tests/test_end_to_end.py::test_mutation_separates_a_test_that_checks_from_one_that_merely_runs— passed, groundedtests/test_end_to_end.py::test_a_run_killed_mid_mutant_is_put_back_by_the_next_one— passed, groundedtests/test_mutation.py::test_the_score_separates_a_test_that_checks_from_one_that_merely_runs— passed, ungroundedtests/test_mutation.py::test_every_test_citing_the_story_gets_a_chance_to_notice— passed, ungroundedtests/test_mutation.py::test_the_operator_set_is_three_source_mutations_applied_to_pythons_own_tree— passed, ungroundedtests/test_mutation.py::test_a_comparison_is_flipped_to_its_negation_not_to_a_neighbour— passed, ungroundedtests/test_mutation.py::test_a_returned_value_is_swapped_for_a_different_one_never_the_same_one— passed, ungroundedtests/test_mutation.py::test_an_interrupted_run_leaves_the_tree_recoverable_and_says_so— passed, ungroundedtests/test_the_jvm_adapter.py::test_a_mutation_that_changes_nothing_is_never_generated— skipped, not-checkedtests/test_the_jvm_adapter.py::test_an_ordinary_board_puts_back_what_an_interrupted_mutation_left— skipped, not-checked
Requirements (11)
What the user needs
- dev07-2-3-ur1every test citing the story runs against each mutant
- dev07-2-3-ur2a run that does not finish inside its bound counts as killed — tests that never complete against broken code are the opposite of passing over it unmoved
- dev07-2-3-ur3the original is parked before the mutant goes down and restored before the next run's tests, and a recovery names the source it restored
What the system does
- dev07-2-3-sr1exactly one site changes per mutant, enumerated from a fresh parse each time, so a survivor means one difference
- dev07-2-3-sr2a mutation that changes nothing is never generated: a comparison flips to its negation, a return swaps for a different value
- dev07-2-3-sr3the mutants are the adapter's own — no third-party engine, no version matrix
- dev07-2-3-sr4mutation edits the real source in place, which is what lets the tests run against the project as it is — its own imports, plugins and configuration
- dev07-2-3-sr5each mutant is loaded the way the project would load it, with nothing held over from the mutant before
How well
- dev07-2-3-nfr1off by default because a mutant costs a test run and a project has thousands; the check finds defects a fully green suite has signed off
Still open
- dev07-2-3-u1where the operator set should stop is a cost and catch tradeoff, and is not fixed
- dev07-2-3-u2while the pass runs, the file on disk is a mutant, and every other reader of the tree — a commit, a CI checkout, an editor — sees it, and what closes that window is unsettled
dev07-2-4 report the score on the story Proven
As a developer reading the result I want a proportion rather than a word, so that a story checked three ways out of forty reads apart from one checked thirty-nine.
The tests that cite it (11)
tests/test_end_to_end.py::test_mutation_separates_a_test_that_checks_from_one_that_merely_runs— passed, groundedtests/test_end_to_end.py::test_mutation_separates_a_test_that_checks_from_one_that_merely_runs— passed, groundedtests/test_end_to_end.py::test_every_python_survivor_is_named— passed, groundedtests/test_end_to_end.py::test_each_survivor_names_the_stories_it_sits_in— passed, groundedtests/test_end_to_end.py::test_a_reach_nobody_measured_is_not_reported_rather_than_zero— passed, groundedtests/test_mutation.py::test_the_score_separates_a_test_that_checks_from_one_that_merely_runs— passed, ungroundedtests/test_mutation.py::test_a_question_never_asked_is_reported_as_nothing_not_as_a_score— passed, ungroundedtests/test_mutation.py::test_survivors_are_advisory_and_cannot_un_prove_a_story— passed, ungroundedtests/test_mutation.py::test_a_score_must_be_a_proportion_that_holds_together— passed, ungroundedtests/test_the_jvm_adapter.py::test_mutation_names_what_survived_and_not_only_how_many— skipped, not-checkedtests/test_the_jvm_adapter.py::test_a_survivor_no_test_reached_is_said_apart_from_one_no_test_noticed— skipped, not-checked
Requirements (9)
What the user needs
- dev07-2-4-ur1the score is killed out of total, on the story, because the mutant faced every citing test
- dev07-2-4-ur2a story that drove nothing wired reports nothing at all, never a zero
- dev07-2-4-ur3each survivor is named — what was changed, and where in the source — because the reader's next move differs per survivor
- dev07-2-4-ur4a survivor no test reached is said apart from one no test noticed
- dev07-2-4-ur5a shortfall is a candidate for triage and never a failure; whether one blocks is the opting team's policy, never the core's
- dev07-2-4-ur6a survivor cannot move a story's status — the gate is the wiring fact, and mutation sits beside it
- dev07-2-4-ur7a count nobody measured is reported as not reported, never as 0: where the run did not say whether a survivor was reached, how many went unreached is not reported, because a 0 there claims every survivor was reached
What the system does
- dev07-2-4-sr1the proportion is checked at the manifest's edge: more killed than run is corrupt, never stricter; more names than shortfall is refused, and fewer is degradation, with the count read over the names
Still open
- dev07-2-4-u1an equivalent mutant is killed by nothing and cannot be told from a real survivor, which is why the score stays advisory