Evaluation

dev · us-east-1
Band agreement75%105 class answers in the run's LOW / MEDIUM / HIGH band
Weighted κ0.64Chance-corrected band agreement, quadratic weights
Mean absolute error1.69|respondent − run| on the 0–10 scale, 77 scored answers
Bias+0.90Respondent minus run; positive means the run under-rates
Within ±142%Scored answers within one point of the run
Responses1514 distinct respondents · 9 on reference cases

Respondent score against run score

One dot per scored answer (77); hover or focus a dot for the case and class. The band lines at 4.0 and 7.0 are the LOW / MEDIUM / HIGH thresholds.

LOWLOWMEDIUMMEDIUMHIGHHIGH0.00.04.04.07.07.010.010.0Run scoreRespondent score22222333422223
Table of the 77 scored answers
SubmittedCaseClassRespondentRunΔBands
2026-10-29 22:03S071 Whose AI brain is this, really?98.2+0.80HIGH / HIGH
2026-10-29 22:03S072 What are we feeding it?96.7+2.30HIGH / MEDIUM
2026-10-29 22:03S073 What does it create from our data: and where does THAT go?97.1+1.90HIGH / HIGH
2026-10-29 22:03S074 Who's deciding what the AI sees and does?107.9+2.10HIGH / HIGH
2026-10-29 22:03S075 Which humans are actually looking at our stuff?105.3+4.70HIGH / MEDIUM
2026-10-29 22:03S076 Who's holding the steering wheel?97.9+1.10HIGH / HIGH
2026-10-29 22:03S077 How many companies are really in this chain: and did we agree to it?107.4+2.60HIGH / HIGH
2026-10-27 15:40S101 Whose AI brain is this, really?89.2−1.20HIGH / HIGH
2026-10-27 15:40S102 What are we feeding it?87.9+0.10HIGH / HIGH
2026-10-27 15:40S103 What does it create from our data: and where does THAT go?77.9−0.90HIGH / HIGH
2026-10-27 15:40S104 Who's deciding what the AI sees and does?88.6−0.60HIGH / HIGH
2026-10-27 15:40S105 Which humans are actually looking at our stuff?55.2−0.20MEDIUM / MEDIUM
2026-10-27 15:40S106 Who's holding the steering wheel?87.9+0.10HIGH / HIGH
2026-10-27 15:40S107 How many companies are really in this chain: and did we agree to it?88.6−0.60HIGH / HIGH
2026-10-26 19:00S061 Whose AI brain is this, really?108.6+1.40HIGH / HIGH
2026-10-26 19:00S062 What are we feeding it?98.3+0.70HIGH / HIGH
2026-10-26 19:00S063 What does it create from our data: and where does THAT go?107.2+2.80HIGH / HIGH
2026-10-26 19:00S064 Who's deciding what the AI sees and does?109.0+1.00HIGH / HIGH
2026-10-26 19:00S065 Which humans are actually looking at our stuff?87.4+0.60HIGH / HIGH
2026-10-26 19:00S066 Who's holding the steering wheel?1010.00.00HIGH / HIGH
2026-10-26 19:00S067 How many companies are really in this chain: and did we agree to it?98.3+0.70HIGH / HIGH
2026-10-26 12:19S081 Whose AI brain is this, really?77.9−0.90HIGH / HIGH
2026-10-26 12:19S082 What are we feeding it?87.3+0.70HIGH / HIGH
2026-10-26 12:19S083 What does it create from our data: and where does THAT go?85.7+2.30HIGH / MEDIUM
2026-10-26 12:19S084 Who's deciding what the AI sees and does?67.3−1.30MEDIUM / HIGH
2026-10-26 12:19S085 Which humans are actually looking at our stuff?85.7+2.30HIGH / MEDIUM
2026-10-26 12:19S086 Who's holding the steering wheel?87.4+0.60HIGH / HIGH
2026-10-26 12:19S087 How many companies are really in this chain: and did we agree to it?97.4+1.60HIGH / HIGH
2026-10-23 15:40S011 Whose AI brain is this, really?65.0+1.00MEDIUM / MEDIUM
2026-10-23 15:40S012 What are we feeding it?75.6+1.40HIGH / MEDIUM
2026-10-23 15:40S013 What does it create from our data: and where does THAT go?75.6+1.40HIGH / MEDIUM
2026-10-23 15:40S014 Who's deciding what the AI sees and does?56.3−1.30MEDIUM / MEDIUM
2026-10-23 15:40S015 Which humans are actually looking at our stuff?43.3+0.70MEDIUM / LOW
2026-10-23 15:40S016 Who's holding the steering wheel?77.8−0.80HIGH / HIGH
2026-10-23 15:40S017 How many companies are really in this chain: and did we agree to it?87.0+1.00HIGH / HIGH
2026-10-22 20:21own-36a57b0f1 Whose AI brain is this, really?23.4−1.40LOW / LOW
2026-10-22 20:21own-36a57b0f2 What are we feeding it?55.4−0.40MEDIUM / MEDIUM
2026-10-22 20:21own-36a57b0f3 What does it create from our data: and where does THAT go?23.0−1.00LOW / LOW
2026-10-22 20:21own-36a57b0f4 Who's deciding what the AI sees and does?12.2−1.20LOW / LOW
2026-10-22 20:21own-36a57b0f5 Which humans are actually looking at our stuff?22.7−0.70LOW / LOW
2026-10-22 20:21own-36a57b0f6 Who's holding the steering wheel?23.6−1.60LOW / LOW
2026-10-22 20:21own-36a57b0f7 How many companies are really in this chain: and did we agree to it?56.3−1.30MEDIUM / MEDIUM
2026-10-22 18:57own-a49512151 Whose AI brain is this, really?97.1+1.90HIGH / HIGH
2026-10-22 18:57own-a49512152 What are we feeding it?96.7+2.30HIGH / MEDIUM
2026-10-22 18:57own-a49512153 What does it create from our data: and where does THAT go?86.0+2.00HIGH / MEDIUM
2026-10-22 18:57own-a49512154 Who's deciding what the AI sees and does?97.7+1.30HIGH / HIGH
2026-10-22 18:57own-a49512155 Which humans are actually looking at our stuff?94.7+4.30HIGH / MEDIUM
2026-10-22 18:57own-a49512156 Who's holding the steering wheel?96.3+2.70HIGH / MEDIUM
2026-10-22 18:57own-a49512157 How many companies are really in this chain: and did we agree to it?96.3+2.70HIGH / MEDIUM
2026-10-22 14:55own-4c5627251 Whose AI brain is this, really?83.2+4.80HIGH / LOW
2026-10-22 14:55own-4c5627252 What are we feeding it?93.4+5.60HIGH / LOW
2026-10-22 14:55own-4c5627253 What does it create from our data: and where does THAT go?93.4+5.60HIGH / LOW
2026-10-22 14:55own-4c5627254 Who's deciding what the AI sees and does?73.9+3.10HIGH / LOW
2026-10-22 14:55own-4c5627255 Which humans are actually looking at our stuff?92.8+6.20HIGH / LOW
2026-10-22 14:55own-4c5627256 Who's holding the steering wheel?73.4+3.60HIGH / LOW
2026-10-22 14:55own-4c5627257 How many companies are really in this chain: and did we agree to it?83.9+4.10HIGH / LOW
2026-10-21 17:50S021 Whose AI brain is this, really?97.9+1.10HIGH / HIGH
2026-10-21 17:50S022 What are we feeding it?76.6+0.40HIGH / MEDIUM
2026-10-21 17:50S023 What does it create from our data: and where does THAT go?98.2+0.80HIGH / HIGH
2026-10-21 17:50S024 Who's deciding what the AI sees and does?97.9+1.10HIGH / HIGH
2026-10-21 17:50S025 Which humans are actually looking at our stuff?75.3+1.70HIGH / MEDIUM
2026-10-21 17:50S026 Who's holding the steering wheel?97.1+1.90HIGH / HIGH
2026-10-21 17:50S027 How many companies are really in this chain: and did we agree to it?98.2+0.80HIGH / HIGH
2026-10-21 15:40S021 Whose AI brain is this, really?87.9+0.10HIGH / HIGH
2026-10-21 15:40S022 What are we feeding it?46.6−2.60MEDIUM / MEDIUM
2026-10-21 15:40S023 What does it create from our data: and where does THAT go?38.2−5.20LOW / HIGH
2026-10-21 15:40S024 Who's deciding what the AI sees and does?87.9+0.10HIGH / HIGH
2026-10-21 15:40S025 Which humans are actually looking at our stuff?72.7+4.30HIGH / LOW
2026-10-21 15:40S026 Who's holding the steering wheel?87.1+0.90HIGH / HIGH
2026-10-21 15:40S027 How many companies are really in this chain: and did we agree to it?78.2−1.20HIGH / HIGH
2026-10-15 14:26S051 Whose AI brain is this, really?55.8−0.80MEDIUM / MEDIUM
2026-10-15 14:26S052 What are we feeding it?45.5−1.50MEDIUM / MEDIUM
2026-10-15 14:26S053 What does it create from our data: and where does THAT go?54.5+0.50MEDIUM / MEDIUM
2026-10-15 14:26S054 Who's deciding what the AI sees and does?56.3−1.30MEDIUM / MEDIUM
2026-10-15 14:26S055 Which humans are actually looking at our stuff?23.8−1.80LOW / LOW
2026-10-15 14:26S056 Who's holding the steering wheel?76.8+0.20HIGH / MEDIUM
2026-10-15 14:26S057 How many companies are really in this chain: and did we agree to it?77.7−0.70HIGH / HIGH

Band agreement matrix

All 105 answers, including the band-only responses from before 2026-10-14

You said ↓ · run said →LOWMEDIUMHIGH
LOW2019%00%11%
MEDIUM22%1615%11%
HIGH88%1413%4341%

Outlined diagonal cells are agreement. Below the diagonal the run rated a class higher than the respondent; above it the respondent rated higher.

If the formula were shifted by this feedback

Scored answers only (77 answers from 11 responses), so the baseline differs from the headline figure

Adjustment to run scoresAgreementMAEAgreement, held outMAE, held out
None (as scored)68%1.6968%1.69
One global shift (+0.90)68%1.5466%1.65
One shift per class71%1.5269%1.67

The first two columns fit the shift on the same answers it is judged on. The held-out columns refit it with each response left out, which is what a new respondent would see.

The formula's level values are the uncalibrated null prior of ADR-0005 §4; a calibrated v2 would carry the SOV:2.0 prefix and is fitted to expert-reviewed vectors, not to survey scores.

Nearest-neighbour calibration

Each answer predicted from the k scored answers with the closest run score (77 answers from 11 responses)

PredictionAgreementMAEAgreement, held outMAE, held out
None (as scored)68%1.6968%1.69
k = 3 neighbours45%2.0344%2.48
k = 5 neighbours60%1.7555%2.26
k = 7 neighbours64%1.7761%2.11

A local alternative to the constant shifts above: the prediction is the mean respondent score of the neighbours, so it can bend differently at the low and high end. Ties go to the same hazard class. In-sample lets the other classes of the same response vote; held out drops the whole response.

With few responses one respondent can supply most of a neighbourhood. The lookup shows how many responses each reading rests on.

Lookup: what respondents said near each run score (k = 5)
Run scoreNeighbours saidBandResponses
14.2MEDIUM / LOW3
24.2MEDIUM / LOW3
35.6MEDIUM / LOW3
44.8MEDIUM / MEDIUM3
57.4HIGH / MEDIUM5
67.2HIGH / MEDIUM2
78.6HIGH / HIGH5
88.2HIGH / HIGH3
98.8HIGH / HIGH2
109.2HIGH / HIGH2

How the figures moved

Cumulative over responses in submission order; the last point is today's headline

Band agreement after each response

0%25%50%75%100%123456789101112131415Response

Mean absolute error after each response

0.01.02.03.04.0123456789101112131415Response
Table of the 15 responses in order
#SubmittedCaseAnswers so farBand agreementMAE
12026-09-30 21:41own-e3c397047100%—
22026-10-07 23:17own-e456fa5914100%—
32026-10-08 17:51own-bec6abe02195%—
42026-10-14 18:06S092896%—
52026-10-15 14:26S053594%0.97
62026-10-21 15:40S024290%1.51
72026-10-21 17:50S024988%1.38
82026-10-22 14:55own-4c5627255677%2.21
92026-10-22 18:57own-a49512156371%2.26
102026-10-22 20:21own-36a57b0f7074%2.07
112026-10-23 15:40S017773%1.93
122026-10-26 12:19S088471%1.86
132026-10-26 19:00S069174%1.77
142026-10-27 15:40S109876%1.64
152026-10-29 22:03S0710575%1.69

The run against the expert reading, and against itself

50 completed runs of 10 reference cases, 2026-09-28 to 2026-10-28. No respondents involved; 10 runs of user-authored cases have no expert reading and are left out.

Headline band84%Runs whose headline lands in the expert reading's band; the headline class matched in 100%
Class band43%350 class scores in the expert reading's LOW / MEDIUM / HIGH band
Carrying dimension65%Classes where the run chose the expert reading's dimension
Metric agreement68%Exact on the 6203 metric cells the expert reading defines; 95% exact or one level off
Score bias+1.79Run minus expert per class; mean absolute error 2.18
Repeat stability80%Metric cells identical across repeat runs of the same case; 90% between any two runs

By reference case

Expert and run headlines on the 0–10 scale; agreement figures pool every class of every run

CaseRunsExpertRun headlinesClass bandDimensionMetrics exactBiasStable cells
S01145.6 MEDIUM5.6 – 6.354%58%70%+0.5271%
S0257.1 HIGH7.1 – 7.931%97%69%+1.9982%
S0344.7 MEDIUM5.6 – 6.386%68%76%+1.1885%
S0413.4 LOW4.114%71%66%+1.49—
S0545.0 MEDIUM5.843%57%64%+2.6984%
S0658.1 HIGH8.1 – 10.054%63%69%+1.7781%
S0747.1 HIGH7.4 – 8.839%46%72%+1.8078%
S0836.6 MEDIUM6.6 – 7.343%52%77%+1.2082%
S0957.4 HIGH8.3 – 9.231%86%68%+2.3282%
S1056.1 MEDIUM7.0 – 8.60%60%46%+4.8377%

Run headlines show the range across that case's runs; a muted range means every run landed in the expert reading's band. Stable cells is the share of the 147 metric cells that took one value across every run of the case, so it needs two runs or more.

The expert reading is each case's reviewed ground truth; agreement is measured against it class by class, as in the S01 pilot write-up.

By metric

Agent-assigned metrics, least stable first. CR, IR and AR come straight from the questionnaire and never vary.

MetricUnstable across runsExact, expert-definedOverrides priorOverride rightPrior right
RE Exit / Mitigation Effort40%76% /5138%0%8%
VI Primary Dependency Impact -- Integrity37%46% /35026%34%23%
AV Access Path35%68% /35040%55%37%
UI Org Action Required33%63% /35031%64%21%
VC Primary Dependency Impact -- Confidentiality32%45% /35057%30%35%
SI Downstream Blast-Radius Impact -- Integrity30%55% /35021%41%18%
E Precedent Maturity27%35% /35026%21%48%
U Vendor Notice Posture27%47% /4349%0%42%
VA Primary Dependency Impact -- Availability25%52% /35015%41%20%
S Regulatory Consequence24%93% /5916%0%44%
SA Downstream Blast-Radius Impact -- Availability22%55% /35025%52%15%
SC Downstream Blast-Radius Impact -- Confidentiality17%41% /35033%24%25%
V Blast-Radius Concentration17%87% /3509%0%100%
PR Position Required14%50% /35022%38%62%
AT Trigger Conditions13%74% /35029%48%52%
R Portability / Exit Recovery13%86% /35010%11%86%
AU Silent/Automatic Trigger5%79% /10049%12%10%
AC Compulsion Complexity3%88% /3508%28%72%

Unstable: share of (case, class) cells with repeat runs where the runs disagreed on this metric. Exact is judged only where the expert reading defines the metric; the readings leave most threat and supplemental metrics undefined while the agents always assign one, and the count after the slash is the cells compared.

Overrides prior: how often the agent left the questionnaire's value. Override right and prior right are, for those same cells, how often each matched the expert reading. Overall the agents overrode 24% of priors, were right 28% of the time, and the prior they left would have been right 33% of the time.

By hazard class

Band agreement compares severity bands; bias is respondent minus run on the 0–10 scale. Mean scores per class are on the Responses page.

ClassAnswersBand agreementBiasMAE
1 Whose AI brain is this, really?1587%+0.621.40
2 What are we feeding it?1567%+0.821.64
3 What does it create from our data: and where does THAT go?1567%+0.932.22
4 Who's deciding what the AI sees and does?1587%+0.271.31
5 Which humans are actually looking at our stuff?1553%+2.012.50
6 Who's holding the steering wheel?1580%+0.791.23
7 How many companies are really in this chain: and did we agree to it?1587%+0.881.57

Who agrees with the run

The same figures split by the respondent's profile and the kind of case

SegmentAnswersBand agreementBiasMAE
Case type
Reference case6379%+0.481.30
Own assessment4269%+2.032.75
CVSS familiarity
Rarely or never (1–2)4967%+0.351.45
Sometimes (3)14100%+1.031.03
Routinely (4–5)4276%+2.152.49
Years in the field
0-24267%+0.221.51
3-514100%——
6-102864%+2.292.29
11-202190%+0.871.37
Team
AI / ML engineering2879%−0.471.30
GRC / compliance14100%+1.031.03
Other2162%+1.611.92
Security engineering / architecture3571%+2.152.49
Third-party / vendor risk771%+1.111.11

Closing statements

Five statements rated 1 (strongly disagree) to 5 (strongly agree) by 15 respondents

Individual responses
The headline rating is right4.3
Next steps are actionable4.5
Clear without technical background4.1
Traceability increases trust4.4
Would use for a real review4.0
Strongly disagree Disagree Neutral Agree Strongly agreeRight-hand number is the mean on the 1–5 scale
Table of counts per level
Statement12345Mean
The headline rating is right001864.3
Next steps are actionable001594.5
Clear without technical background013564.1
Traceability increases trust002584.4
Would use for a real review004744.0