Plumb

Specification

dev07-1 · run the suite under observation

A sub-flow of dev07 · run plumb. Entered when plumb is run, carrying the configuration, the annotations and the working tree. Exits to dev07-3 · collect what the run did, carrying the run, a trace per citing test, and a wiring fact each.

3 of 6 stories proven · 0 passed, not wired · 0 unproven · 3 with nothing holding them

dev07-1-1 check the runtime is the build's Proven

As a developer whose build is green I want plumb's run to be the build's run, so that the board reports my code rather than the observation.

The tests that cite it (16)
  • tests/test_adoption.py::test_each_build_tool_is_told_how_to_produce_its_own_classpath — passed, ungrounded
  • tests/test_adoption.py::test_a_reactor_module_missing_from_the_classpath_refuses_the_run — passed, ungrounded
  • tests/test_adoption.py::test_a_classpath_with_the_build_outputs_out_of_order_is_refused — passed, ungrounded
  • tests/test_adoption.py::test_the_remedy_prepends_in_the_order_the_build_runs — passed, ungrounded
  • tests/test_adoption.py::test_a_module_answered_by_an_installed_jar_says_so — passed, ungrounded
  • tests/test_adoption.py::test_a_run_wider_than_the_build_is_not_refused — passed, ungrounded
  • tests/test_adoption.py::test_a_project_with_no_build_tool_is_asked_nothing — passed, ungrounded
  • tests/test_adoption.py::test_the_check_can_be_turned_off_and_says_that_it_was — passed, ungrounded
  • tests/test_adoption.py::test_gradle_resources_left_off_the_classpath_refuse_the_run — passed, ungrounded
  • tests/test_adoption.py::test_a_gradle_subproject_missing_entirely_is_named — passed, ungrounded
  • tests/test_adoption.py::test_the_kotlin_dsl_declares_projects_the_same_way — passed, ungrounded
  • tests/test_adoption.py::test_a_gradle_build_with_everything_on_the_classpath_passes — passed, ungrounded
  • tests/test_adoption.py::test_an_output_directory_the_build_never_wrote_is_not_demanded — passed, ungrounded
  • tests/test_artifact.py::test_a_project_shadowing_the_tool_leaves_this_process — passed, ungrounded
  • tests/test_artifact.py::test_the_project_is_imported_before_the_artifact — passed, ungrounded
  • tests/test_end_to_end.py::test_a_board_refuses_rather_than_observing_a_run_the_build_does_not_perform — passed, grounded
Requirements (9)
What the user needs
  • dev07-1-1-ur1the adapter's runtime matches the project's own — for a JVM project, the classpath
  • dev07-1-1-ur2a run missing something the build has is refused by name, before the suite is launched
  • dev07-1-1-ur3a run wider than the build's is noted and runs
What the system does
  • dev07-1-1-sr1the adapter borrows the build's own runtime rather than reconstructing it, and each build tool is told how to produce its own classpath
  • dev07-1-1-sr2order is part of the runtime: the run's order is the build's, and equal members in another order are refused
  • dev07-1-1-sr3the diagnosis names the missing module alone; the remedy is the whole classpath in the build's order, ready to use, never an instruction to add to what is there
  • dev07-1-1-sr4a module answered by an installed artifact of itself is said as that, and never as a missing directory
  • dev07-1-1-sr5the check reads a declared build tool; a project with none is asked nothing
How well
  • dev07-1-1-nfr1a refusal says how to proceed anyway, and what proceeding costs

dev07-1-2 check which version wrote the tree's state Nothing holds it

As a developer who has just upgraded I want to be told when my tree still holds the last version's derived state, so that I read a board the upgrade explains.

Requirements (5)
What the user needs
  • dev07-1-2-ur1a version difference is said before the board is read, and names the remedy
  • dev07-1-2-ur2plumb offers to clear what it wrote; what the build tool wrote is named and never deleted
What the system does
  • dev07-1-2-sr1plumb stamps its version on everything it writes
  • dev07-1-2-sr2the stamp is compared against the running version before the board renders
How well
  • dev07-1-2-nfr1a version difference is a condition and never a refusal

dev07-1-3 trace each citing test Proven

As an adopter I want the evidence gathered by the language I already run, so that adopting this costs no coverage tool and no version matrix.

The tests that cite it (29)
  • tests/test_adapter_parity.py::test_the_python_recorder_draws_members_of_one_module — passed, ungrounded
  • tests/test_adapter_parity.py::test_both_adapters_draw_the_same_graph_over_the_same_program — skipped, not-checked
  • tests/test_adoption.py::test_an_instrument_that_reached_nothing_says_so_rather_than_reporting_a_clean_run — passed, untouched
  • tests/test_board.py::test_a_board_whose_observer_died_says_that_and_not_declare_entry_points — passed, ungrounded
  • tests/test_config.py::test_a_project_declares_itself_in_one_file_whatever_language_it_is — passed, ungrounded
  • tests/test_config.py::test_the_declared_values_stay_in_the_adapters_own_spelling — passed, ungrounded
  • tests/test_config.py::test_sources_default_to_production_where_the_two_are_the_same_question — passed, ungrounded
  • tests/test_end_to_end.py::test_a_blind_run_still_boards_and_the_board_says_the_instrument_failed — passed, grounded
  • tests/test_frame_identity.py::test_no_observed_name_carries_a_segment_the_runtime_invented — passed, untouched
  • tests/test_frame_identity.py::test_the_names_observed_are_exactly_the_definitions_declared — passed, untouched
  • tests/test_frame_identity.py::test_the_edges_are_named_by_the_same_rule — passed, untouched
  • tests/test_frame_identity.py::test_the_reconstruction_agrees_with_a_runtime_that_reports_the_name — passed, ungrounded
  • tests/test_grounding.py::test_a_project_declares_its_surface_in_config_not_in_its_source — passed, ungrounded
  • tests/test_grounding.py::test_a_project_that_declares_nothing_degrades_rather_than_erroring — passed, ungrounded
  • tests/test_the_jvm_adapter.py::test_every_probe_the_advice_calls_is_one_the_flag_declares — passed, untouched
  • tests/test_the_jvm_adapter.py::test_the_bootstrap_probe_compiles_to_one_class_and_no_other — skipped, not-checked
  • tests/test_the_jvm_adapter.py::test_the_jvm_adapter_runs_a_suite_and_reports_what_it_watched — skipped, not-checked
  • tests/test_the_jvm_adapter.py::test_the_window_closes_when_the_entry_point_throws — skipped, not-checked
  • tests/test_the_jvm_adapter.py::test_a_method_containing_a_switch_survives_the_weave — skipped, not-checked
  • tests/test_the_jvm_adapter.py::test_an_entry_point_inside_the_surface_is_mutated_like_any_other_production_code — skipped, not-checked
  • tests/test_the_jvm_adapter.py::test_a_long_run_says_where_it_has_got_to — skipped, not-checked
  • tests/test_the_jvm_adapter.py::test_the_heartbeat_estimates_from_the_position_it_was_given — passed, ungrounded
  • tests/test_the_jvm_adapter.py::test_the_position_names_the_unit_the_phase_actually_counts — passed, ungrounded
  • tests/test_the_jvm_adapter.py::test_a_member_the_toolchain_made_is_not_an_end_of_an_edge — passed, untouched
  • tests/test_trace.py::test_trace_records_production_frames_in_call_order — passed, untouched
  • tests/test_trace.py::test_trace_ignores_code_outside_the_production_surface — passed, untouched
  • tests/test_trace.py::test_reading_the_artifact_as_data_is_observed_without_executing_it — passed, untouched
  • tests/test_trace.py::test_the_tracer_restores_whatever_tracing_was_already_in_place — passed, untouched
  • tests/test_trace.py::test_a_production_root_may_be_a_single_file — passed, ungrounded
Requirements (21)
What the user needs
  • dev07-1-3-ur01the trace is per test, taken with the runtime's own hooks
  • dev07-1-3-ur02the declared entry points are few, stable, and the project's own — never a story-to-code map
  • dev07-1-3-ur08a frame is named for the source definition it belongs to — the enclosing function, method or module — so a name means the same thing on every runtime of its language
  • dev07-1-3-ur09an expression a runtime may or may not give a frame of its own — a comprehension, a lambda, a generator expression — is named for the definition containing it
  • dev07-1-3-ur10one suite observed under two runtimes of one language names one set of frames, so a series spanning a runtime upgrade compares like with like
  • dev07-1-3-ur11one source names one set of frames whichever toolchain built the artifact, so a series spanning a compiler upgrade compares like with like
What the system does
  • dev07-1-3-sr01where watching is unaffordable, the production surface reports its own execution, at a cost that depends on neither the surface size nor the shape of the test
  • dev07-1-3-sr02the observation runs in the process it happens in: no second process, no attach, no debugger
  • dev07-1-3-sr03what is observed, where, and what is recorded are declared apart from the mechanism that puts an observation in place, and that mechanism knows none of them
  • dev07-1-3-sr04what is production and what is a test is decided by where its code was loaded from, never by its name
  • dev07-1-3-sr05the window follows the work to a thread it started and a task it queued, and is kept per thread
  • dev07-1-3-sr08one naming rule serves the whole adapter: what a trace records, what an entry point is matched against and what mutation may scope to are the same names
  • dev07-1-3-sr09a frame with no source to place it in keeps the name its runtime gave it
  • dev07-1-3-sr10a member the artifact declares as made by its toolchain — by the flags it carries or by the declaration naming what encloses it — is named for the source member that contains it, and never by matching what it is called
How well
  • dev07-1-3-nfr1running under observation costs about what the tests already cost
  • dev07-1-3-nfr2an instrument failure costs the wiring fact and never the run; the board says the instrument failed, not the project
  • dev07-1-3-nfr3code the adapter cannot instrument is left exactly as it was, and named
Still open
  • dev07-1-3-u1what tracing costs on Python tracks the traced window rather than the codebase, and no bound is claimable
  • dev07-1-3-u2a runtime-generated class reports the origin of its host, so every one is instrumented, and what that costs on a large suite is unmeasured
  • dev07-1-3-u3a runtime-made class that declares no enclosing member can be placed only in its host, and whether the host is a sufficient name is unsettled
  • dev07-1-3-u4a thread that predates the window and takes work by accepting a connection is seen by neither adapter; a corpus that needs those threads needs a collector outside them

dev07-1-5 say how far the run has got Nothing holds it

As a developer waiting on a long suite I want to see how far it has got, so that waiting is a decision I make rather than one made for me.

Requirements (2)
What the user needs
  • dev07-1-5-ur1the run says where it has got to: tests done out of tests discovered, coarsely
  • dev07-1-5-ur2the position is readable at any moment without following the run's own output

dev07-1-4 name what each test did Proven

As a developer reading PROVEN I want it to mean the test went in through an entry point, so that reaching the code means reaching it the way anybody would.

The tests that cite it (38)
  • tests/test_adapter_conformance.py::test_the_python_adapter_answers_the_shared_scenario[drives a declared entry point] — passed, ungrounded
  • tests/test_adapter_conformance.py::test_the_python_adapter_answers_the_shared_scenario[runs production under no entry point] — passed, ungrounded
  • tests/test_adapter_conformance.py::test_the_python_adapter_answers_the_shared_scenario[reads the artifact and runs none of it] — passed, ungrounded
  • tests/test_adapter_conformance.py::test_the_python_adapter_answers_the_shared_scenario[neither runs nor reads the artifact] — passed, ungrounded
  • tests/test_adapter_conformance.py::test_the_java_adapter_answers_the_shared_scenario[drives a declared entry point] — failed, ungrounded
  • tests/test_adapter_conformance.py::test_the_java_adapter_answers_the_shared_scenario[runs production under no entry point] — failed, ungrounded
  • tests/test_adapter_conformance.py::test_the_java_adapter_answers_the_shared_scenario[reads the artifact and runs none of it] — failed, ungrounded
  • tests/test_adapter_conformance.py::test_the_java_adapter_answers_the_shared_scenario[neither runs nor reads the artifact] — failed, ungrounded
  • tests/test_adapter_conformance.py::test_the_core_reads_back_every_fact_an_adapter_may_report — passed, untouched
  • tests/test_adoption.py::test_a_failed_helper_reports_why_and_not_only_that_it_failed — passed, ungrounded
  • tests/test_board.py::test_the_board_says_what_it_could_not_check_and_names_the_code — passed, ungrounded
  • tests/test_config.py::test_a_java_project_whose_modules_compile_separately_names_every_root — passed, ungrounded
  • tests/test_config.py::test_a_tests_key_that_is_not_a_path_is_refused_by_name — passed, ungrounded
  • tests/test_end_to_end.py::test_with_no_entry_points_declared_the_board_says_so — passed, grounded
  • tests/test_end_to_end.py::test_declaring_entry_points_separates_wired_proof_from_isolated_proof — passed, grounded
  • tests/test_end_to_end.py::test_orphaned_code_cannot_ground_however_green_its_test_is — passed, grounded
  • tests/test_gate.py::test_no_entry_points_is_surfaced_with_its_code_never_silently_skipped — passed, ungrounded
  • tests/test_gate.py::test_a_grounded_story_surfaces_no_condition — passed, ungrounded
  • tests/test_grounding.py::test_every_value_the_fact_can_take_is_derived_from_the_trace — passed, ungrounded
  • tests/test_grounding.py::test_inertness_is_asked_first_because_it_needs_no_entry_points — passed, ungrounded
  • tests/test_grounding.py::test_inert_needs_roots_not_merely_the_absence_of_entry_points — passed, ungrounded
  • tests/test_grounding.py::test_orphaned_code_cannot_ground — passed, ungrounded
  • tests/test_grounding.py::test_dispatched_is_not_a_weaker_not_checked — passed, ungrounded
  • tests/test_grounding.py::test_a_story_lost_two_different_ways_reports_both_reasons — passed, ungrounded
  • tests/test_grounding.py::test_a_coarse_dispatch_signal_never_weakens_the_orphan_check — passed, ungrounded
  • tests/test_grounding.py::test_a_test_that_grounds_and_also_reaches_orphaned_code_says_so — passed, ungrounded
  • tests/test_grounding.py::test_the_tracer_tells_a_mixed_test_from_a_clean_one — passed, untouched
  • tests/test_the_jvm_adapter.py::test_a_test_that_touches_no_production_code_reads_untouched — skipped, not-checked
  • tests/test_the_jvm_adapter.py::test_a_surface_nobody_watches_is_not_named_to_the_adapter — passed, ungrounded
  • tests/test_the_jvm_adapter.py::test_inert_is_withheld_when_nothing_instrumented_the_surface — skipped, not-checked
  • tests/test_trace.py::test_execution_under_a_declared_entry_point_is_distinguishable — passed, untouched
  • tests/test_trace.py::test_without_declared_entry_points_nothing_can_ground — passed, ungrounded
  • tests/test_trace.py::test_a_test_that_touches_nothing_is_visible_as_such — passed, untouched
  • tests/test_trace.py::test_a_per_thread_tracer_may_not_claim_inert_while_threads_are_unaccounted_for — passed, ungrounded
  • tests/test_trace.py::test_a_per_thread_tracer_may_not_claim_inert_while_threads_are_unaccounted_for — passed, ungrounded
  • tests/test_trace.py::test_inert_survives_when_there_is_nothing_to_account_for — passed, ungrounded
  • tests/test_trace.py::test_the_trace_names_the_outermost_door_and_not_the_one_reached_on_the_way — passed, ungrounded
  • tests/test_trace.py::test_a_test_driving_two_entrances_names_both_and_one_that_grounds_by_none_names_none — passed, ungrounded
Requirements (16)
What the user needs
  • dev07-1-4-ur1the wiring fact per test is one of five: dev07-1-4-ur1.1 - grounded — it drove production code inside an entry point's window dev07-1-4-ur1.2 - ungrounded — it ran production code outside every entry point dev07-1-4-ur1.3 - inert — it read the artifact and ran nothing dev07-1-4-ur1.4 - not-checked — no entry points are declared, so the question could not be asked * dev07-1-4-ur1.5 - dispatched — the work left for a thread that could not be followed; the hand-off was seen, never assumed
  • dev07-1-4-ur2inert is asserted only where every thread was accounted for; otherwise the fact is dispatched
  • dev07-1-4-ur3the entry points a test entered are named — the outermost, so declaring another entry point never rewrites the fact for a test that did not change
  • dev07-1-4-ur4a test that enters an entry point and also runs code no entry point reaches says so
  • dev07-1-4-ur5dispatched and not-checked stay separate answers, because the reader's next move differs
  • dev07-1-4-ur6with no entry points declared the board prompts — declare entry points to catch orphaned code — never a silent skip
  • dev07-1-4-ur7an entry point is matched against the names a trace records, so a declared entry point holds whichever runtime observed it
What the system does
  • dev07-1-4-sr1grounded is read from the per-thread window: production code is recorded only while the window is open
  • dev07-1-4-sr2a hand-off is observed on the side that knows the window: a started thread is marked by its starter, and a queued task carries the window and releases it on every exit path
  • dev07-1-4-sr3an entry point is recorded when entered while no window is open, which is what makes the fact the outermost
  • dev07-1-4-sr4thread accounting is sampled at the window's two ends, and both of its error directions read as dispatched — the answer that can cost a status and never grant one
How well
  • dev07-1-4-nfr1a fact that cannot be gathered degrades and surfaces; the run never fails for it
Still open
  • dev07-1-4-u1what counts as a production frame is the adapter's to settle, and a cold and a warm run may report different facts
  • dev07-1-4-u2whether a declared entry point is legitimate cannot be checked, only recorded — an entry point opened for one subject is open to every subject
  • dev07-1-4-u3a test that runs the tool in another process reads as having run nothing; whether the hand-off is the right fact for a process boundary is open
  • dev07-1-4-u4static dead-code detection — code reachable from no entry point regardless of any test — is a separate question, undecided

dev07-1-6 measure the surface the citing tests reached Nothing holds it

As a developer deciding where the next test goes I want to see which of the declared surface my citing tests never reached through an entry point, so that the gap is something I can point at.

Requirements (7)
What the user needs
  • dev07-1-6-ur1what the instrument knew of and did not reach is reported beside what it reached; the unreached part is the measurement
  • dev07-1-6-ur2the figure is the declared surface this run's citing tests drove through an entry point, never the lines a suite executed, and is named so the two are not compared
  • dev07-1-6-ur3the figure is asked for, never on by default, and moves no status
What the system does
  • dev07-1-6-sr1an instrument that cannot enumerate its own sites says the figure could not be measured, as any other question this method could not ask is said
  • dev07-1-6-sr2the figure is kept beside the manifest by the observer that made it and enters neither the manifest nor the board, so no interface widens to carry it
How well
  • dev07-1-6-nfr1it costs a run nothing it does not already pay: a site's ends are settled before the program starts, and reaching one is already recorded
Still open
  • dev07-1-6-u1what it costs an instrument that must enumerate its sites by reading source, rather than by keeping what it instrumented, is unmeasured

← Every chapter