CodeMap Remote — measuring prompt quality in an AI system

Every number on this page is recomputed from an artifact committed in the repository: nightly quality runs, judge calibrations, the personas' nights, partition decisions with their drift, and prompt-optimisation runs. No model runs to build it. Built 2026-10-22T09:19:15Z. The live counterpart is the set of public Grafana Cloud dashboards below; the served MCP is https://codemap.checkitout.app/mcp.

The latest night — 2026-09-20

Grounded answers

94 %
judge ≥ 4 on grounding, share of judged answers

Correct answers

90 %
judge ≥ 4 on correctness

Mean rating

3.9
personas rate after verifying a pointer; unverified ratings cap at 3

Judge vs oracle

κ 0.45 · AC1 0.90
Cohen's κ and Gwet's AC1 of the judge's 'located' against the execution oracle on 24 rows; agreement 92 %, oracle yes-rate 92 %; gate 0.6 on κ, or on AC1 with agreement ≥ 80 % when the yes-rate is past 85 % (κ's prevalence paradox); basis: ac1

Credits per correct answer

15.6
credits are relative units, never currency

Version drift

31 %
bank rows answering differently after the last pack decision, not invalidated on purpose

Answers judged

154
166 asks, 36 misses reported, 1 budget refusals

Honest abstention

98 %
all 26 %, false 2 %

Night over night

Grounded

94 %
5 night(s)

Correct

90 %
5 night(s)

Mean rating (of 5)

3.9
5 night(s)

Pointer-verified ratings

97 %
5 night(s)

Version drift

31 %
5 night(s)

Credits per correct answer

15.6
5 night(s)

Who asked — the synthetic users

personaconversationsturnscreditsmean ratingmisses
haiku-ops12229201.74.71
haiku-pm12168206.94.52
opus-architect374369.13.64
opus-reviewer8222551.33.514
sonnet-bugfixer17265288.03.312
sonnet-newcomer19228226.14.42

What it cost — in credits and tokens, never currency

Credits by user

opus-reviewer551.3
opus-architect369.1
judge327.0
sonnet-bugfixer288.0
sonnet-newcomer226.1
haiku-pm206.9
haiku-ops201.7

Tokens

cached31,492,243
cache_creation1,107,073
completion242,951
prompt1,232
cache-read ratio 97 % · p95 latency 47,550 ms

Gain campaign (pairs across nights)

35 / 30 pairs
tokens ratio 0.63 · turns Δ -3.5 · seconds Δ -88.5 · rated at least as well 29 % · fewer tokens 17 %; the row flips at thirty pairs

Gains vs no-CodeMap baselines (this night)

31 pairs
tokens ratio 0.64 · turns Δ -3.9 · seconds Δ -94.2; a gain without its baseline row is ungated

The graph follows the code — decisions on the pull request

packrepo @ headdecisionbyplacednew subsystemsFAQ invalidateddriftreclued
1.0.1backend @ ff43730acceptRamzesX7JUnit test-execution harness106/55 (11 %)—
1.1.0backend+frontend @ ff43730acceptRamzesX174codemap-trajectory-viewer1217/54 (31 %)0, 3, 4, 6, 8, 9, 10, 11, 16, 170, 171, 174, 175, 176, 178, 205, 206

The prompt improves on request — GEPA runs

runmodewhereseed (val)best (val)candidatesmetric callsoutcome
2026-09-16-confirm-candidatedry-runbox1.00——12seed measured
2026-09-16-confirm-seeddry-runbox0.92——12seed measured
2026-09-16-modal-drydry-runmodal0.67——3seed measured
2026-09-16-noproposalgepabox0.830.83130no win — recorded
2026-09-16bgepabox0.831.00333promoted candidate
promptbuiltparentwhygate
v12026-09-16—seed: the API navigator prompt's protocol + the big tier's graph lesson, rewritten for pointersn/a (baseline)
v22026-09-16v1GEPA run 2026-09-16b: val 0.8333333333333334 → 1.0 on 6 examples, 33 metric callspromoted (0.03 gate)

Live dashboards (Grafana Cloud, public)

Visual proofs

2026 09 16 ai system dashboard
2026 09 16 ai system dashboard
2026 09 16 issue 4 decided
2026 09 16 issue 4 decided
2026 09 16 issue 4 proposal
2026 09 16 issue 4 proposal
2026 09 16 pages quality
2026 09 16 pages quality