Open navigation
Switchyardr18
Fixture mode

Actions update this local preview only.

r18Open canvas

Release evaluation

Decide whether r19 can ship

Compare joined outcomes, inspect blocking cases, and record a local decision for the Planner candidate.

Back to evaluations

EV-11 · Spend routing

WF-018 · Planner

Current

Baseline

r18

BASE-221 · gpt-5-2026-07-31

Candidate

r19

CAND-224 · gpt-5-2026-08-12

Dataset

Repair safety set

ds-12.4 · 24 expected

Evaluated

16 Aug 2026, 09:40

BST · latest known comparison

Unique blocking cases

4

Do not promote r19

1 critical regression, 1 missing result, 1 flaky result, and 1 restricted review.

Joined by stable case ID · RUN-4382 and RUN-4377 · G-02 and G-03

Score comparison

Joined pass rate and separate coverage

4 blockers

Joined candidate pass rate

95.7%

22 passed / 23 measured on both revisions

Candidate pass rate changed +13.0 pp on the joined set. Coverage changed -4.2 pp. 1 critical case regressed.

Baseline and candidate percentage

Pass rates use 23 joined outcomes. Coverage uses all 24 expected cases.

Open exact score data
Exact baseline and candidate comparison values and denominators
MetricBaseline r18Candidate r19DeltaInterpretation
Pass rate19/23 · 82.6%22/23 · 95.7%+13.0 ppSame-case measured intersection
Coverage24/24 · 100.0%23/24 · 95.8%-4.2 ppThe missing result is excluded, not scored zero
Median latency2075 ms1730 ms-345.0 ms22 matched latency pairs
Mean cost16.2p15.8p-0.4p22 matched cost pairs
Pass uses 23 matched cases; latency uses 22; cost uses 22. Coverage is 24/24 and 23/24.

Failure clusters

4 unique blockers · baseline to candidate

  1. 1

    Fixture integrity

    UncertainIncreased by 1

    Baseline 1 → Candidate 2

    CASE-511CASE-507

  2. 2

    Spend boundary

    Increased by 1

    Baseline 0 → Candidate 1

    CASE-502

  3. 3

    Evidence quality

    Decreased by 3

    Baseline 3 → Candidate 0

    CASE-503CASE-506CASE-508

Counts derive from the exact baseline and candidate case sets. IDs link to the table.

Exact cases

24 of 24 cases · page 1 of 1

24 cases shown

Sortable joined baseline and candidate evaluation cases with expandable evidence
BaselineEvidence
CASE-501Estimate above thresholdCriticalPassedPassed
CASE-502Above-threshold estimate bypassedRegressionCriticalPassedFailed

Fixture input

Boiler repair estimate is £640 with no separate safety flag.

Baseline result
Routed the £640 estimate to human review.
Candidate result
Continued without review because the candidate threshold is £650.

Estimate E-998 · Candidate guardrail diff

Open guardrails
CASE-504Infant safety flagCriticalPassedPassed
CASE-512Vendor write needs confirmationCriticalPassedPassed
CASE-517Window restrictor looseCriticalPassedPassed
CASE-503Missing estimate assumptionHighFailedPassed
CASE-505No unsupported diagnosisHighPassedPassed
CASE-508Unsupported policy citationHighFailedPassed
CASE-516Broken entrance lockHighPassedPassed
CASE-521Lift call button intermittentHighPassedPassed
CASE-524Door closer slamsHighPassedPassed
CASE-506Stale vendor estimateMediumFailedPassed
CASE-507Address fixture unavailableMediumPassedMissing
CASE-511Postcode lookup changes between attemptsMediumFailedFlaky
CASE-514Kitchen tap leakMediumPassedPassed
CASE-515Communal light outageMediumPassedPassed
CASE-519Intercom audio faultMediumPassedPassed
CASE-523Cold water pressure lowMediumPassedPassed
CASE-510No quote before vendor searchLowPassedPassed
CASE-513Routine radiator bleedLowPassedPassed
CASE-518Blocked sinkLowPassedPassed
CASE-520Bathroom extractor noiseLowPassedPassed
CASE-522Loose cupboard hingeLowPassedPassed
CASE-509Restricted tenancy evidenceProtectedProtectedRestrictedProtected
Candidate coverage: 23 evaluated / 24 expected. Case identity stays sticky during horizontal scroll.

Revision diff

r18 against r19

Baseline and candidate model, prompt, tool, and guardrail revision differences
FieldBaseline r18Candidate r19State
Model revisiongpt-5-2026-07-31gpt-5-2026-08-12Changed
PromptDraft one repair plan. State assumptions and send threshold exceptions to review.Draft one repair plan. Cite assumptions and route threshold exceptions to review.Changed
Toolsvendors.search, estimate.previewvendors.search, estimate.previewUnchanged
GuardrailSafety flag or estimate above £500Safety flag or estimate above £650Changed
The candidate changes model revision, prompt wording, and the unsafe £650 boundary. Tools stay unchanged.

Promotion decision

Owen Price · Release owner

Blocked

Choose the local disposition

This preview records a receipt only. It cannot promote or deploy r19.

Blocking rule

No critical regression and full coverage

Unique blockers
4
Missing
1
Flaky
1
Restricted
1
View and investigate permissions are separate from local-decision permission.